[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:15:06 dist-test-slave-vl24 krb5kdc[5460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:15:06 dist-test-slave-vl24 krb5kdc[5460](info): set up 2 sockets
Sep 23 20:15:06 dist-test-slave-vl24 krb5kdc[5460](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 20:15:07.610185 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:15:07 dist-test-slave-vl24 krb5kdc[5460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:15:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:15:07.786779 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:41439
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:08.143992 5473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:08.144352 5473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:08.144460 5473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:08.151722 5473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:08.151845 5473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:08.151892 5473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:08.151942 5473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:08.151988 5473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:15:08.162525 5473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:41439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:08.164425 5473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:08.166139 5473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:08.185828 5478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:08.185873 5479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:08.185935 5481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:08.188987 5473 server_base.cc:1034] running on GCE node
I20260923 20:15:08.190130 5473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:08.192351 5473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:08.193579 5473 hybrid_clock.cc:648] HybridClock initialized: now 1790194508193486 us; error 82 us; skew 500 ppm
I20260923 20:15:08.194064 5473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:08 dist-test-slave-vl24 krb5kdc[5460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194508, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:08.197485 5473 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:15:08.200006 5473 webserver.cc:460] Webserver started at http://127.5.83.126:45007/ using document root <none> and password file <none>
I20260923 20:15:08.201392 5473 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:08.201582 5473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:08.202011 5473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:08.206310 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "cdd0b363ef684da69d0f2c2856b015e9"
format_stamp: "Formatted at 2026-09-23 20:15:08 on dist-test-slave-vl24"
server_key: "c6a29ae62358c1c6b9e7ce1e3c11a569"
server_key_iv: "26332d0b3dd7864c70cc3c20c76ab631"
server_key_version: "encryptionkey@0"
I20260923 20:15:08.207398 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "cdd0b363ef684da69d0f2c2856b015e9"
format_stamp: "Formatted at 2026-09-23 20:15:08 on dist-test-slave-vl24"
server_key: "c6a29ae62358c1c6b9e7ce1e3c11a569"
server_key_iv: "26332d0b3dd7864c70cc3c20c76ab631"
server_key_version: "encryptionkey@0"
I20260923 20:15:08.214617 5473 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260923 20:15:08.220116 5488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:08.222231 5473 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260923 20:15:08.222528 5473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "cdd0b363ef684da69d0f2c2856b015e9"
format_stamp: "Formatted at 2026-09-23 20:15:08 on dist-test-slave-vl24"
server_key: "c6a29ae62358c1c6b9e7ce1e3c11a569"
server_key_iv: "26332d0b3dd7864c70cc3c20c76ab631"
server_key_version: "encryptionkey@0"
I20260923 20:15:08.222766 5473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:08.254514 5473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:08.259931 5473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:08.260298 5473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:08.284235 5473 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:41439
I20260923 20:15:08.284263 5539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:41439 every 8 connection(s)
I20260923 20:15:08.286586 5473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:08.292873 5540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:08.296638 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 5473
I20260923 20:15:08.296973 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:08.299139 5453 external_mini_cluster.cc:1428] Setting key ec88b0cc0972ebec93cde434163b8f43
I20260923 20:15:08.310045 5540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9: Bootstrap starting.
I20260923 20:15:08.315088 5540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:08.317206 5540 log.cc:824] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:08.322139 5540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9: No bootstrap required, opened a new log
Sep 23 20:15:08 dist-test-slave-vl24 krb5kdc[5460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:08.334802 5540 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd0b363ef684da69d0f2c2856b015e9" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41439 } }
I20260923 20:15:08.335457 5540 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:08.335695 5540 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd0b363ef684da69d0f2c2856b015e9, State: Initialized, Role: FOLLOWER
I20260923 20:15:08.337237 5540 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd0b363ef684da69d0f2c2856b015e9" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41439 } }
I20260923 20:15:08.337584 5540 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:08.337873 5540 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:08.338162 5540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:08.341818 5540 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd0b363ef684da69d0f2c2856b015e9" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41439 } }
I20260923 20:15:08.342976 5540 leader_election.cc:302] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdd0b363ef684da69d0f2c2856b015e9; no voters:
I20260923 20:15:08.343806 5540 leader_election.cc:288] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:08.344063 5545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:08.347306 5540 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:08.348511 5542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:08.305182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.5.83.126:41439)
0923 20:15:08.306331 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:08.306365 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:08.307550 (+ 1185us) server_negotiation.cc:407] Connection header received
0923 20:15:08.317322 (+ 9772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:08.317343 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:08.318031 (+ 688us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:08.318608 (+ 577us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:08.323698 (+ 5090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:08.325269 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:08.328069 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:08.328952 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:08.329141 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:08.336832 (+ 7691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:08.337071 (+ 239us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:08.337121 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:08.337185 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:08.341381 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:08.342322 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:08.342330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:08.342339 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:08.342663 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:08.343125 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:08.343131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:08.343134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:08.344062 (+ 928us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:08.344493 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:08.346001 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0923 20:15:08.346522 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":406,"threads_started":1}
I20260923 20:15:08.348565 5545 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [term 1 LEADER]: Becoming Leader. State: Replica: cdd0b363ef684da69d0f2c2856b015e9, State: Running, Role: LEADER
I20260923 20:15:08.349645 5545 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd0b363ef684da69d0f2c2856b015e9" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41439 } }
I20260923 20:15:08.357105 5547 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd0b363ef684da69d0f2c2856b015e9. Latest consensus state: current_term: 1 leader_uuid: "cdd0b363ef684da69d0f2c2856b015e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd0b363ef684da69d0f2c2856b015e9" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41439 } } }
I20260923 20:15:08.357578 5547 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:08.357209 5546 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdd0b363ef684da69d0f2c2856b015e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd0b363ef684da69d0f2c2856b015e9" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41439 } } }
I20260923 20:15:08.358117 5546 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9 [sys.catalog]: This master's current role is: LEADER
W20260923 20:15:08.366706 5561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:15:08.366890 5561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:15:08.367046 5557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:08.369141 5557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:08.380656 5557 catalog_manager.cc:1347] Generated new cluster ID: 4bb53930e12843e0ba661380be9e52ad
I20260923 20:15:08.380784 5557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:08.401189 5557 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:08.403443 5557 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:08.411957 5557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdd0b363ef684da69d0f2c2856b015e9: Generated new TSK 0
I20260923 20:15:08.413280 5557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:15:08.456650 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41439
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:08.792500 5566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:08.792820 5566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:08.792922 5566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:08.800304 5566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:08.800420 5566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:08.800561 5566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:15:08.810616 5566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41439
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:08.811978 5566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:08.813494 5566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:08.827549 5572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:08.827467 5574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:08.827620 5571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:08.828768 5566 server_base.cc:1034] running on GCE node
I20260923 20:15:08.829411 5566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:08.830315 5566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:08.831549 5566 hybrid_clock.cc:648] HybridClock initialized: now 1790194508831516 us; error 47 us; skew 500 ppm
I20260923 20:15:08.831871 5566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:08 dist-test-slave-vl24 krb5kdc[5460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194508, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:08.839483 5566 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:15:08.841291 5566 webserver.cc:460] Webserver started at http://127.5.83.65:33599/ using document root <none> and password file <none>
I20260923 20:15:08.842047 5566 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:08.842267 5566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:08.842690 5566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:08.845911 5566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "aca08b1e19ba4a6e8a8dc0b7d5aa1b8b"
format_stamp: "Formatted at 2026-09-23 20:15:08 on dist-test-slave-vl24"
server_key: "5605584355f57e65a44c86462db9c58c"
server_key_iv: "e7e9a80605cc946361ce74e608ccc48c"
server_key_version: "encryptionkey@0"
I20260923 20:15:08.846750 5566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "aca08b1e19ba4a6e8a8dc0b7d5aa1b8b"
format_stamp: "Formatted at 2026-09-23 20:15:08 on dist-test-slave-vl24"
server_key: "5605584355f57e65a44c86462db9c58c"
server_key_iv: "e7e9a80605cc946361ce74e608ccc48c"
server_key_version: "encryptionkey@0"
I20260923 20:15:08.853088 5566 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 20:15:08.856654 5581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:08.858103 5566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:15:08.858294 5566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "aca08b1e19ba4a6e8a8dc0b7d5aa1b8b"
format_stamp: "Formatted at 2026-09-23 20:15:08 on dist-test-slave-vl24"
server_key: "5605584355f57e65a44c86462db9c58c"
server_key_iv: "e7e9a80605cc946361ce74e608ccc48c"
server_key_version: "encryptionkey@0"
I20260923 20:15:08.858500 5566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:08.888811 5566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:08.893005 5566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:08.893317 5566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:08.894347 5566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:08.896015 5566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:08.896122 5566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:08.896323 5566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:08.896413 5566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:08.931730 5566 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:35225
I20260923 20:15:08.931787 5693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:35225 every 8 connection(s)
I20260923 20:15:08.933125 5566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:15:08.934698 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 5566
I20260923 20:15:08.934979 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:15:08.935792 5453 external_mini_cluster.cc:1428] Setting key 7c2f72697fdf544f8e66ac6c0793efa6
Sep 23 20:15:08 dist-test-slave-vl24 krb5kdc[5460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194508, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
I20260923 20:15:08.957068 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:08.936741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:59819 (local address 127.5.83.126:41439)
0923 20:15:08.937230 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:08.937236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:08.938314 (+ 1078us) server_negotiation.cc:407] Connection header received
0923 20:15:08.940165 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:08.940170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:08.940259 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:08.940374 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:08.942060 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:08.942783 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:08.945455 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:08.945879 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:08.945985 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:08.951045 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:08.951050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:08.951054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:08.951088 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:08.953790 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:08.954590 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:08.954595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:08.954597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:08.954721 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:08.955120 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:08.955124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:08.955125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:08.955401 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:08.955515 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:08.956447 (+ 932us) server_negotiation.cc:299] Negotiation successful
0923 20:15:08.956736 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":247,"threads_started":1}
I20260923 20:15:08.958518 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:08.937324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41439 (local address 127.5.83.65:59819)
0923 20:15:08.938159 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:08.938214 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:08.939903 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:08.940676 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:08.940684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:08.941295 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:08.941857 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:08.941874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:08.942935 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:08.942943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:08.945279 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:08.945298 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:08.946124 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:08.946134 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:08.946456 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:08.947267 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:08.947304 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:08.950734 (+ 3430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:08.953929 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:08.953936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:08.953950 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:08.954460 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:08.954822 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:08.954825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:08.954827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:08.955009 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:08.955520 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:08.955557 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:08.955936 (+ 379us) client_negotiation.cc:769] Sending connection context
0923 20:15:08.956266 (+ 330us) client_negotiation.cc:240] Negotiation successful
0923 20:15:08.956898 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":337,"threads_started":1}
I20260923 20:15:08.960278 5694 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41439
I20260923 20:15:08.960861 5694 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:08.961946 5694 heartbeater.cc:511] Master 127.5.83.126:41439 requested a full tablet report, sending...
I20260923 20:15:08.966089 5505 ts_manager.cc:194] Registered new tserver with Master: aca08b1e19ba4a6e8a8dc0b7d5aa1b8b (127.5.83.65:35225)
I20260923 20:15:08.968073 5505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:59819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:15:08.979995 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41439
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:09.343998 5700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:09.344275 5700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:09.344441 5700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:09.352577 5700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:09.352774 5700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:09.353015 5700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:15:09.363370 5700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41439
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:09.364879 5700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:09.366339 5700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:09.381738 5708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:09.381681 5706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:09.381698 5705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:09.383195 5700 server_base.cc:1034] running on GCE node
I20260923 20:15:09.383916 5700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:09.385176 5700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:09.386425 5700 hybrid_clock.cc:648] HybridClock initialized: now 1790194509386389 us; error 68 us; skew 500 ppm
I20260923 20:15:09.386801 5700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:09 dist-test-slave-vl24 krb5kdc[5460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194509, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:09.390355 5700 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:15:09.393484 5700 webserver.cc:460] Webserver started at http://127.5.83.66:39899/ using document root <none> and password file <none>
I20260923 20:15:09.394248 5700 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:09.394388 5700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:09.394755 5700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:09.397897 5700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "029fda80fa724deda2d5beeacf23e1af"
format_stamp: "Formatted at 2026-09-23 20:15:09 on dist-test-slave-vl24"
server_key: "7577af2296bb2012fc48b69e3b457602"
server_key_iv: "7dbb369df1bffd0b2edc6a1a0a020c70"
server_key_version: "encryptionkey@0"
I20260923 20:15:09.398795 5700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "029fda80fa724deda2d5beeacf23e1af"
format_stamp: "Formatted at 2026-09-23 20:15:09 on dist-test-slave-vl24"
server_key: "7577af2296bb2012fc48b69e3b457602"
server_key_iv: "7dbb369df1bffd0b2edc6a1a0a020c70"
server_key_version: "encryptionkey@0"
I20260923 20:15:09.404381 5700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:15:09.408241 5715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:09.409817 5700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:09.410090 5700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "029fda80fa724deda2d5beeacf23e1af"
format_stamp: "Formatted at 2026-09-23 20:15:09 on dist-test-slave-vl24"
server_key: "7577af2296bb2012fc48b69e3b457602"
server_key_iv: "7dbb369df1bffd0b2edc6a1a0a020c70"
server_key_version: "encryptionkey@0"
I20260923 20:15:09.410326 5700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:09.434429 5700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:09.439232 5700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:09.439548 5700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:09.440646 5700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:09.442164 5700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:09.442292 5700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:09.442446 5700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:09.442514 5700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:09.481292 5700 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:40953
I20260923 20:15:09.481366 5827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:40953 every 8 connection(s)
I20260923 20:15:09.482596 5700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:15:09.486528 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 5700
I20260923 20:15:09.486727 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:15:09.487254 5453 external_mini_cluster.cc:1428] Setting key 5f5d8508bc910a38d6629cb4116f5c28
Sep 23 20:15:09 dist-test-slave-vl24 krb5kdc[5460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194509, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
I20260923 20:15:09.503906 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:09.485266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:47207 (local address 127.5.83.126:41439)
0923 20:15:09.485673 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:09.485678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:09.486887 (+ 1209us) server_negotiation.cc:407] Connection header received
0923 20:15:09.488815 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:09.488820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:09.488909 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:09.489036 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:09.490499 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:09.491266 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:09.492895 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:09.493330 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:09.493471 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:09.497802 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:09.497807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:09.497811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:09.497847 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:09.500754 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:09.501523 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:09.501534 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:09.501536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:09.501659 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:09.502106 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:09.502110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:09.502112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:09.502372 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:09.502529 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:09.503296 (+ 767us) server_negotiation.cc:299] Negotiation successful
0923 20:15:09.503540 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":199,"threads_started":1}
I20260923 20:15:09.505018 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:09.485834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41439 (local address 127.5.83.66:47207)
0923 20:15:09.486690 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:09.486753 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:09.488500 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:09.489267 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:09.489274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:09.489783 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:09.490332 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:09.490349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:09.491436 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:09.491439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:09.492750 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:09.492762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:09.493566 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:09.493574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:09.493804 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:09.494489 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:09.494527 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:09.497561 (+ 3034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:09.500914 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:09.500923 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:09.500943 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:09.501382 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:09.501781 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:09.501786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:09.501789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:09.501975 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:09.502509 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:09.502527 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:09.502880 (+ 353us) client_negotiation.cc:769] Sending connection context
0923 20:15:09.503205 (+ 325us) client_negotiation.cc:240] Negotiation successful
0923 20:15:09.503765 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":397,"threads_started":1}
I20260923 20:15:09.506469 5828 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41439
I20260923 20:15:09.506805 5828 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:09.507463 5828 heartbeater.cc:511] Master 127.5.83.126:41439 requested a full tablet report, sending...
I20260923 20:15:09.508896 5505 ts_manager.cc:194] Registered new tserver with Master: 029fda80fa724deda2d5beeacf23e1af (127.5.83.66:40953)
I20260923 20:15:09.510383 5505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:47207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
I20260923 20:15:09.972419 5694 heartbeater.cc:503] Master 127.5.83.126:41439 was elected leader, sending a full tablet report...
I20260923 20:15:10.513003 5828 heartbeater.cc:503] Master 127.5.83.126:41439 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
W20260923 20:15:10.655990 5453 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.83.67: real 1.166s user 0.000s sys 0.008s
I20260923 20:15:10.656795 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41439
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:10.990844 5834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:10.991128 5834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:10.991281 5834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:10.998688 5834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:10.998809 5834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:10.999014 5834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:15:11.009056 5834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41439
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:11.010461 5834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:11.011853 5834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:11.025732 5839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:11.025785 5842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:11.025810 5840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:11.026680 5834 server_base.cc:1034] running on GCE node
I20260923 20:15:11.027432 5834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:11.028667 5834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:11.029867 5834 hybrid_clock.cc:648] HybridClock initialized: now 1790194511029814 us; error 43 us; skew 500 ppm
I20260923 20:15:11.030207 5834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:11 dist-test-slave-vl24 krb5kdc[5460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194511, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:11.033428 5834 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:15:11.034922 5834 webserver.cc:460] Webserver started at http://127.5.83.67:38709/ using document root <none> and password file <none>
I20260923 20:15:11.035600 5834 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:11.035699 5834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:11.035962 5834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:11.039101 5834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "912019ee32cd461eafa54bc58e363c29"
format_stamp: "Formatted at 2026-09-23 20:15:11 on dist-test-slave-vl24"
server_key: "f089fb7bcf7432956c0c087a98884978"
server_key_iv: "68f751cf4ec413bf14f16ada568bba47"
server_key_version: "encryptionkey@0"
I20260923 20:15:11.039834 5834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "912019ee32cd461eafa54bc58e363c29"
format_stamp: "Formatted at 2026-09-23 20:15:11 on dist-test-slave-vl24"
server_key: "f089fb7bcf7432956c0c087a98884978"
server_key_iv: "68f751cf4ec413bf14f16ada568bba47"
server_key_version: "encryptionkey@0"
I20260923 20:15:11.045768 5834 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260923 20:15:11.049541 5849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:11.051023 5834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:15:11.051275 5834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "912019ee32cd461eafa54bc58e363c29"
format_stamp: "Formatted at 2026-09-23 20:15:11 on dist-test-slave-vl24"
server_key: "f089fb7bcf7432956c0c087a98884978"
server_key_iv: "68f751cf4ec413bf14f16ada568bba47"
server_key_version: "encryptionkey@0"
I20260923 20:15:11.051510 5834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:11.066401 5834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:11.070520 5834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:11.070899 5834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:11.071944 5834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:11.073508 5834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:11.073657 5834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:11.073807 5834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:11.073900 5834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:11.109391 5834 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:36565
I20260923 20:15:11.109438 5961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:36565 every 8 connection(s)
I20260923 20:15:11.110667 5834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:15:11.113474 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 5834
I20260923 20:15:11.113682 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:15:11.114264 5453 external_mini_cluster.cc:1428] Setting key daa3d151e55e18bf46262250b2a26352
Sep 23 20:15:11 dist-test-slave-vl24 krb5kdc[5460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194511, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:11.131654 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:11.113379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:49651 (local address 127.5.83.126:41439)
0923 20:15:11.113974 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:11.113983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:11.115072 (+ 1089us) server_negotiation.cc:407] Connection header received
0923 20:15:11.117314 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:11.117320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:11.117402 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:11.117591 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:11.119190 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.120004 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.121629 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.122038 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.122144 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:11.125601 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:11.125606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:11.125609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:11.125640 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:11.128096 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:11.128877 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:11.128881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:11.128884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:11.129012 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:11.129541 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:11.129546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:11.129548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:11.129853 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:11.130015 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:11.130989 (+ 974us) server_negotiation.cc:299] Negotiation successful
0923 20:15:11.131207 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":302,"threads_started":1}
I20260923 20:15:11.132519 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:11.113988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41439 (local address 127.5.83.67:49651)
0923 20:15:11.114888 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:11.114939 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:11.117029 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:11.117760 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:11.117767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:11.118273 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:11.118985 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:11.119012 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.120136 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.120140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:11.121492 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:11.121503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.122237 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.122242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:11.122428 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:11.123044 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:11.123066 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:11.125413 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:11.128219 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:11.128226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:11.128240 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:11.128764 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:11.129128 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:11.129132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:11.129135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:11.129393 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:11.129996 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:11.130002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:11.130470 (+ 468us) client_negotiation.cc:769] Sending connection context
0923 20:15:11.130795 (+ 325us) client_negotiation.cc:240] Negotiation successful
0923 20:15:11.131306 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":317,"threads_started":1}
I20260923 20:15:11.133914 5962 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41439
I20260923 20:15:11.134243 5962 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:11.134958 5962 heartbeater.cc:511] Master 127.5.83.126:41439 requested a full tablet report, sending...
I20260923 20:15:11.136348 5505 ts_manager.cc:194] Registered new tserver with Master: 912019ee32cd461eafa54bc58e363c29 (127.5.83.67:36565)
I20260923 20:15:11.137535 5505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:49651
I20260923 20:15:11.143885 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:15:11 dist-test-slave-vl24 krb5kdc[5460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194511, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:15:11 dist-test-slave-vl24 krb5kdc[5460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194511, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:11.191391 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:11.175973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.5.83.65:35225)
0923 20:15:11.177000 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:11.177007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:11.177033 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:15:11.177093 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:11.177099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:11.177350 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:11.177542 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:11.178970 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.179975 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.181362 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.181940 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.182097 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:11.186124 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:11.186134 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:11.186148 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:11.186179 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:11.188831 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:11.189495 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:11.189503 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:11.189505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:11.189658 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:11.189991 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:11.189994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:11.189996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:11.190329 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:11.190569 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:11.190793 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 20:15:11.190988 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":678,"threads_started":1}
W20260923 20:15:11.192859 5609 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44928
Sep 23 20:15:11 dist-test-slave-vl24 krb5kdc[5460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:15:11 dist-test-slave-vl24 krb5kdc[5460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:11.235706 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:11.221777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.5.83.65:35225)
0923 20:15:11.222029 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:11.222038 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:11.222321 (+ 283us) server_negotiation.cc:407] Connection header received
0923 20:15:11.222542 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:11.222546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:11.222640 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:11.222754 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:11.224070 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.224820 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.226037 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.226457 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.226592 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:11.230558 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:11.230564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:11.230568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:11.230618 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:11.233369 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:11.233994 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:11.233997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:11.234000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:11.234126 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:11.234435 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:11.234438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:11.234440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:11.234682 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:11.234855 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:11.235110 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 20:15:11.235358 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:15:11.237541 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 5566
I20260923 20:15:11.250061 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:11.238445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.5.83.65:35225)
0923 20:15:11.238603 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:11.238607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:11.239031 (+ 424us) server_negotiation.cc:407] Connection header received
0923 20:15:11.239207 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:11.239210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:11.239264 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:11.239340 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:11.240768 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.241448 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.242618 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.243007 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.243104 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:11.245234 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:11.245238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:11.245241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:11.245268 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:11.247892 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:11.248464 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:11.248468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:11.248470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:11.248655 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:11.248989 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:11.248994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:11.249003 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:11.249273 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:11.249443 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:11.249675 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 20:15:11.249815 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 20:15:11.250785 5689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:11.339921 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 5566
I20260923 20:15:11.357872 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 5700
Sep 23 20:15:11 dist-test-slave-vl24 krb5kdc[5460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:15:11.374414 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:11.358803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.5.83.66:40953)
0923 20:15:11.359350 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:11.359358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:11.359384 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:15:11.359449 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:11.359456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:11.359668 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:11.359804 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:11.361055 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.362066 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.363313 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.363857 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.363949 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:11.368518 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:11.368574 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:11.368598 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:11.368638 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:11.371537 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:11.372178 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:11.372185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:11.372187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:11.372309 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:11.372776 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:11.372782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:11.372787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:11.373127 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:11.373457 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:11.373810 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 20:15:11.374080 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":277,"threads_started":1}
I20260923 20:15:11.375803 5823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:11.459016 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 5700
I20260923 20:15:11.477989 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 5834
Sep 23 20:15:11 dist-test-slave-vl24 krb5kdc[5460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:15:11.494513 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:11.479048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.5.83.67:36565)
0923 20:15:11.479676 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:11.479684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:11.479713 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:15:11.479775 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:11.479781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:11.480009 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:11.480139 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:11.481440 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.482465 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.483707 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:11.484285 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:11.484383 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:11.488768 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:11.488781 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:11.488796 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:11.488833 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:11.491447 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:11.492136 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:11.492144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:11.492151 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:11.492272 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:11.492865 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:11.492872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:11.492875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:11.493237 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:11.493555 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:11.493876 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 20:15:11.494163 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":320,"threads_started":1}
I20260923 20:15:11.495931 5957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:11.579859 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 5834
I20260923 20:15:11.596004 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 5473
I20260923 20:15:11.597293 5535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:11.700462 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 5473
2026-09-23T20:15:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5204 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:15:11 dist-test-slave-vl24 krb5kdc[5984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:15:11 dist-test-slave-vl24 krb5kdc[5984](info): set up 2 sockets
Sep 23 20:15:11 dist-test-slave-vl24 krb5kdc[5984](info): commencing operation
krb5kdc: starting...
W20260923 20:15:12.789718 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:15:12 dist-test-slave-vl24 krb5kdc[5984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:15:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:15:12.964233 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:35485
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:33469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:35485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:13.282361 5997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:13.282787 5997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:13.283066 5997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:13.291446 5997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:13.291555 5997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:13.291643 5997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:13.291733 5997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:13.291817 5997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:15:13.302057 5997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:33469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:35485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:35485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:13.303414 5997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:13.304719 5997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:13.316465 6003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:13.316742 6002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:13.316854 6005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:13.317445 5997 server_base.cc:1034] running on GCE node
I20260923 20:15:13.318264 5997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:13.319753 5997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:13.320981 5997 hybrid_clock.cc:648] HybridClock initialized: now 1790194513320951 us; error 56 us; skew 500 ppm
I20260923 20:15:13.321352 5997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:13 dist-test-slave-vl24 krb5kdc[5984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194513, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:13.324486 5997 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:15:13.326010 5997 webserver.cc:460] Webserver started at http://127.5.83.126:41555/ using document root <none> and password file <none>
I20260923 20:15:13.326720 5997 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:13.326864 5997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:13.327188 5997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:13.330268 5997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "e2f1ae95be9844aaa2d69e56c61c254f"
format_stamp: "Formatted at 2026-09-23 20:15:13 on dist-test-slave-vl24"
server_key: "b355e22e2e5d9473b062cf519ae46e4c"
server_key_iv: "8f3215a81f63720e0a8eea1b7274355e"
server_key_version: "encryptionkey@0"
I20260923 20:15:13.331032 5997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "e2f1ae95be9844aaa2d69e56c61c254f"
format_stamp: "Formatted at 2026-09-23 20:15:13 on dist-test-slave-vl24"
server_key: "b355e22e2e5d9473b062cf519ae46e4c"
server_key_iv: "8f3215a81f63720e0a8eea1b7274355e"
server_key_version: "encryptionkey@0"
I20260923 20:15:13.336123 5997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:15:13.339564 6012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:13.341066 5997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:13.341280 5997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "e2f1ae95be9844aaa2d69e56c61c254f"
format_stamp: "Formatted at 2026-09-23 20:15:13 on dist-test-slave-vl24"
server_key: "b355e22e2e5d9473b062cf519ae46e4c"
server_key_iv: "8f3215a81f63720e0a8eea1b7274355e"
server_key_version: "encryptionkey@0"
I20260923 20:15:13.341480 5997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:13.373625 5997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:13.377704 5997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:13.377982 5997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:13.397233 5997 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:35485
I20260923 20:15:13.397269 6063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:35485 every 8 connection(s)
I20260923 20:15:13.398665 5997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:13.400859 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 5997
I20260923 20:15:13.401063 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:13.401723 5453 external_mini_cluster.cc:1428] Setting key 997fc8040477be599a48e57bb0ce4466
I20260923 20:15:13.402776 6064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:15:13 dist-test-slave-vl24 krb5kdc[5984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:13.416895 6064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f: Bootstrap starting.
I20260923 20:15:13.421125 6064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:13.422364 6064 log.cc:824] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:13.425416 6067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:13.403817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44738 (local address 127.5.83.126:35485)
0923 20:15:13.404656 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:13.404673 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:13.404724 (+ 51us) server_negotiation.cc:407] Connection header received
0923 20:15:13.406469 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:13.406499 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:13.407009 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:13.407477 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:13.408874 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:13.410910 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:13.412161 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:13.413205 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:13.413393 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:13.417377 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:13.417387 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:13.417407 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:13.417455 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:13.421364 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:13.421974 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:13.421982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:13.421989 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:13.422206 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:13.422618 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:13.422623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:13.422628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:13.423316 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:13.423670 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:13.424126 (+ 456us) server_negotiation.cc:299] Negotiation successful
0923 20:15:13.424455 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":335,"threads_started":1}
I20260923 20:15:13.425529 6064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f: No bootstrap required, opened a new log
I20260923 20:15:13.432819 6064 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f1ae95be9844aaa2d69e56c61c254f" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 35485 } }
I20260923 20:15:13.433213 6064 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:13.433322 6064 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f1ae95be9844aaa2d69e56c61c254f, State: Initialized, Role: FOLLOWER
I20260923 20:15:13.433888 6064 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f1ae95be9844aaa2d69e56c61c254f" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 35485 } }
I20260923 20:15:13.434136 6064 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:13.434293 6064 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:13.434468 6064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:13.436689 6064 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f1ae95be9844aaa2d69e56c61c254f" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 35485 } }
I20260923 20:15:13.437242 6064 leader_election.cc:302] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2f1ae95be9844aaa2d69e56c61c254f; no voters:
I20260923 20:15:13.437846 6064 leader_election.cc:288] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:13.438004 6069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:13.439147 6069 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [term 1 LEADER]: Becoming Leader. State: Replica: e2f1ae95be9844aaa2d69e56c61c254f, State: Running, Role: LEADER
I20260923 20:15:13.439728 6069 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f1ae95be9844aaa2d69e56c61c254f" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 35485 } }
I20260923 20:15:13.440114 6064 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:13.442651 6071 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2f1ae95be9844aaa2d69e56c61c254f. Latest consensus state: current_term: 1 leader_uuid: "e2f1ae95be9844aaa2d69e56c61c254f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f1ae95be9844aaa2d69e56c61c254f" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 35485 } } }
I20260923 20:15:13.442979 6071 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:13.443459 6070 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2f1ae95be9844aaa2d69e56c61c254f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2f1ae95be9844aaa2d69e56c61c254f" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 35485 } } }
I20260923 20:15:13.443747 6070 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:13.443912 6076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:13.451982 6076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:13.461494 6076 catalog_manager.cc:1347] Generated new cluster ID: 262244985d8f43149878817c5c0bdca7
I20260923 20:15:13.461616 6076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:13.497253 6076 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:13.499269 6076 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:13.509433 6076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2f1ae95be9844aaa2d69e56c61c254f: Generated new TSK 0
I20260923 20:15:13.510613 6076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:15:13.568154 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:35485
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:33469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:13.921499 6090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:13.921770 6090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:13.921931 6090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:13.929574 6090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:13.929683 6090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:13.929891 6090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:15:13.939675 6090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:33469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:35485
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:13.941076 6090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:13.942436 6090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:13.956231 6096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:13.956203 6098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:13.956178 6095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:13.957013 6090 server_base.cc:1034] running on GCE node
I20260923 20:15:13.957660 6090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:13.958590 6090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:13.959836 6090 hybrid_clock.cc:648] HybridClock initialized: now 1790194513959777 us; error 50 us; skew 500 ppm
I20260923 20:15:13.960172 6090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:13 dist-test-slave-vl24 krb5kdc[5984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194513, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:13.967664 6090 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:15:13.969169 6090 webserver.cc:460] Webserver started at http://127.5.83.65:39473/ using document root <none> and password file <none>
I20260923 20:15:13.969841 6090 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:13.969937 6090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:13.970201 6090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:13.973313 6090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "6d6237dadd074ca6b080cbb1f1091251"
format_stamp: "Formatted at 2026-09-23 20:15:13 on dist-test-slave-vl24"
server_key: "36dc76c65a3f40d9874c8f6b1bc63df2"
server_key_iv: "d1c3eff826a620ddfc7b97188fb06fdf"
server_key_version: "encryptionkey@0"
I20260923 20:15:13.974098 6090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "6d6237dadd074ca6b080cbb1f1091251"
format_stamp: "Formatted at 2026-09-23 20:15:13 on dist-test-slave-vl24"
server_key: "36dc76c65a3f40d9874c8f6b1bc63df2"
server_key_iv: "d1c3eff826a620ddfc7b97188fb06fdf"
server_key_version: "encryptionkey@0"
I20260923 20:15:13.980037 6090 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 20:15:13.983640 6105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:13.985128 6090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:13.985303 6090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "6d6237dadd074ca6b080cbb1f1091251"
format_stamp: "Formatted at 2026-09-23 20:15:13 on dist-test-slave-vl24"
server_key: "36dc76c65a3f40d9874c8f6b1bc63df2"
server_key_iv: "d1c3eff826a620ddfc7b97188fb06fdf"
server_key_version: "encryptionkey@0"
I20260923 20:15:13.985515 6090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:14.011762 6090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:14.016029 6090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:14.016322 6090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:14.017346 6090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:14.018682 6090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:14.018751 6090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:14.018888 6090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:14.018951 6090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:14.053918 6090 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:39749
I20260923 20:15:14.053952 6217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:39749 every 8 connection(s)
I20260923 20:15:14.055290 6090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:15:14.058816 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 6090
I20260923 20:15:14.059052 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:15:14.059834 5453 external_mini_cluster.cc:1428] Setting key 1cf65cec70156af3ad66a54131ec17d8
Sep 23 20:15:14 dist-test-slave-vl24 krb5kdc[5984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194513, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
I20260923 20:15:14.076877 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:14.057838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:49087 (local address 127.5.83.126:35485)
0923 20:15:14.058303 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:14.058308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:14.059572 (+ 1264us) server_negotiation.cc:407] Connection header received
0923 20:15:14.061807 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:14.061812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:14.061904 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:14.062017 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:14.063545 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:14.064759 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:14.066358 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:14.066762 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:14.066908 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:14.070971 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:14.070976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:14.070979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:14.071012 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:14.073695 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:14.074455 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:14.074468 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:14.074472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:14.074594 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:14.075034 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:14.075041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:14.075044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:14.075319 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:14.075476 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:14.076227 (+ 751us) server_negotiation.cc:299] Negotiation successful
0923 20:15:14.076450 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":214,"threads_started":1}
I20260923 20:15:14.077960 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:14.058487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:35485 (local address 127.5.83.65:49087)
0923 20:15:14.059374 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:14.059420 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:14.061444 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:14.062226 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:14.062234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:14.062781 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:14.063364 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:14.063381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:14.064926 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:14.064930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:14.066203 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:14.066215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:14.066992 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:14.067001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:14.067320 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:14.068034 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:14.068067 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:14.070741 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:14.073845 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:14.073856 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:14.073874 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:14.074321 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:14.074720 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:14.074725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:14.074729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:14.074918 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:14.075456 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:14.075464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:14.075811 (+ 347us) client_negotiation.cc:769] Sending connection context
0923 20:15:14.076149 (+ 338us) client_negotiation.cc:240] Negotiation successful
0923 20:15:14.076743 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":368,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
I20260923 20:15:14.079756 6218 heartbeater.cc:347] Connected to a master server at 127.5.83.126:35485
I20260923 20:15:14.080083 6218 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:14.080873 6218 heartbeater.cc:511] Master 127.5.83.126:35485 requested a full tablet report, sending...
I20260923 20:15:14.083050 6029 ts_manager.cc:194] Registered new tserver with Master: 6d6237dadd074ca6b080cbb1f1091251 (127.5.83.65:39749)
I20260923 20:15:14.085355 6029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:49087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:15:14.102098 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:35485
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:33469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:14.423514 6224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:14.423766 6224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:14.423856 6224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:14.430990 6224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:14.431102 6224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:14.431216 6224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:15:14.440965 6224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:33469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:35485
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:14.442328 6224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:14.443634 6224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:14.456847 6229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:14.456900 6232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:14.456944 6230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:14.461884 6224 server_base.cc:1034] running on GCE node
I20260923 20:15:14.462446 6224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:14.463310 6224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:14.464602 6224 hybrid_clock.cc:648] HybridClock initialized: now 1790194514464467 us; error 52 us; skew 500 ppm
I20260923 20:15:14.464895 6224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:14 dist-test-slave-vl24 krb5kdc[5984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194514, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:14.467865 6224 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:15:14.469424 6224 webserver.cc:460] Webserver started at http://127.5.83.66:35311/ using document root <none> and password file <none>
I20260923 20:15:14.470124 6224 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:14.470252 6224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:14.470568 6224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:14.473560 6224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "475dd05cb5d2491ebc76ed18091ceef3"
format_stamp: "Formatted at 2026-09-23 20:15:14 on dist-test-slave-vl24"
server_key: "8989d9d9dcb63a7b3e44f249f1023192"
server_key_iv: "103ecca426740a275e93bf6f645329f4"
server_key_version: "encryptionkey@0"
I20260923 20:15:14.474347 6224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "475dd05cb5d2491ebc76ed18091ceef3"
format_stamp: "Formatted at 2026-09-23 20:15:14 on dist-test-slave-vl24"
server_key: "8989d9d9dcb63a7b3e44f249f1023192"
server_key_iv: "103ecca426740a275e93bf6f645329f4"
server_key_version: "encryptionkey@0"
I20260923 20:15:14.479449 6224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:15:14.482795 6239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:14.484117 6224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:15:14.484282 6224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "475dd05cb5d2491ebc76ed18091ceef3"
format_stamp: "Formatted at 2026-09-23 20:15:14 on dist-test-slave-vl24"
server_key: "8989d9d9dcb63a7b3e44f249f1023192"
server_key_iv: "103ecca426740a275e93bf6f645329f4"
server_key_version: "encryptionkey@0"
I20260923 20:15:14.484475 6224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:14.504531 6224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:14.508450 6224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:14.508774 6224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:14.509752 6224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:14.511132 6224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:14.511206 6224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:14.511286 6224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:14.511330 6224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:14.543838 6224 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:46865
I20260923 20:15:14.543882 6351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:46865 every 8 connection(s)
I20260923 20:15:14.545239 6224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:15:14.549798 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 6224
I20260923 20:15:14.549990 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:15:14.550420 5453 external_mini_cluster.cc:1428] Setting key a3a3f3f3f69c1051146ed863db281bb8
Sep 23 20:15:14 dist-test-slave-vl24 krb5kdc[5984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194514, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:14.566854 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:14.547802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:37703 (local address 127.5.83.126:35485)
0923 20:15:14.547973 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:14.547978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:14.549115 (+ 1137us) server_negotiation.cc:407] Connection header received
0923 20:15:14.551201 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:14.551207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:14.551305 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:14.551408 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:14.553096 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:14.553919 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:14.555471 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:14.555902 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:14.556032 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:14.559916 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:14.559925 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:14.559932 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:14.559978 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:14.563541 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:14.564274 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:14.564281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:14.564287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:14.564455 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:14.564913 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:14.564921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:14.564926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:14.565321 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:14.565488 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:14.566221 (+ 733us) server_negotiation.cc:299] Negotiation successful
0923 20:15:14.566458 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
I20260923 20:15:14.567610 6354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:14.548221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:35485 (local address 127.5.83.66:37703)
0923 20:15:14.548956 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:14.548994 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:14.550882 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:14.551786 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:14.551795 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:14.552306 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:14.552930 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:14.552948 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:14.554077 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:14.554082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:14.555327 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:14.555338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:14.556113 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:14.556120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:14.556342 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:14.557138 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:14.557159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:14.559696 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:14.563689 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:14.563696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:14.563707 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:14.564122 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:14.564614 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:14.564617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:14.564619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:14.564789 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:14.565467 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:14.565473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:14.565798 (+ 325us) client_negotiation.cc:769] Sending connection context
0923 20:15:14.566039 (+ 241us) client_negotiation.cc:240] Negotiation successful
0923 20:15:14.566426 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":322,"threads_started":1}
I20260923 20:15:14.569064 6352 heartbeater.cc:347] Connected to a master server at 127.5.83.126:35485
I20260923 20:15:14.569365 6352 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
I20260923 20:15:14.570214 6352 heartbeater.cc:511] Master 127.5.83.126:35485 requested a full tablet report, sending...
I20260923 20:15:14.571798 6029 ts_manager.cc:194] Registered new tserver with Master: 475dd05cb5d2491ebc76ed18091ceef3 (127.5.83.66:46865)
I20260923 20:15:14.573109 6029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:37703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:15:14.591907 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:35485
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:33469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:14.913069 6357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:14.913389 6357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:14.913560 6357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:14.921020 6357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:14.921167 6357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:14.921339 6357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:15:14.931169 6357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:33469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:35485
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:14.932504 6357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:14.933898 6357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:14.946838 6362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:14.946799 6365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:14.947072 6363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:14.948257 6357 server_base.cc:1034] running on GCE node
I20260923 20:15:14.948884 6357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:14.949797 6357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:14.951156 6357 hybrid_clock.cc:648] HybridClock initialized: now 1790194514951089 us; error 54 us; skew 500 ppm
I20260923 20:15:14.951517 6357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:14 dist-test-slave-vl24 krb5kdc[5984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194514, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:14.954694 6357 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:15:14.956161 6357 webserver.cc:460] Webserver started at http://127.5.83.67:40451/ using document root <none> and password file <none>
I20260923 20:15:14.956943 6357 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:14.957077 6357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:14.957389 6357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:14.960261 6357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "932e3e8fefb04e2788293c2e40d14d27"
format_stamp: "Formatted at 2026-09-23 20:15:14 on dist-test-slave-vl24"
server_key: "f3d0b7f9412afee9b6dab0d9dfbeda6a"
server_key_iv: "9834c822eb8c0497fdae011ea6e8b95f"
server_key_version: "encryptionkey@0"
I20260923 20:15:14.961097 6357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "932e3e8fefb04e2788293c2e40d14d27"
format_stamp: "Formatted at 2026-09-23 20:15:14 on dist-test-slave-vl24"
server_key: "f3d0b7f9412afee9b6dab0d9dfbeda6a"
server_key_iv: "9834c822eb8c0497fdae011ea6e8b95f"
server_key_version: "encryptionkey@0"
I20260923 20:15:14.966199 6357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:15:14.969740 6372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:14.971122 6357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:15:14.971295 6357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "932e3e8fefb04e2788293c2e40d14d27"
format_stamp: "Formatted at 2026-09-23 20:15:14 on dist-test-slave-vl24"
server_key: "f3d0b7f9412afee9b6dab0d9dfbeda6a"
server_key_iv: "9834c822eb8c0497fdae011ea6e8b95f"
server_key_version: "encryptionkey@0"
I20260923 20:15:14.971469 6357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:15.010622 6357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:15.014794 6357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:15.015085 6357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:15.016119 6357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:15.017553 6357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:15.017635 6357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:15.017782 6357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:15.017832 6357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:15.050956 6357 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:36917
I20260923 20:15:15.051008 6484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:36917 every 8 connection(s)
I20260923 20:15:15.052268 6357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:15:15.059681 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 6357
I20260923 20:15:15.059857 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:15:15.060385 5453 external_mini_cluster.cc:1428] Setting key d9fa9dd36b00d4c39cf09af3f594f040
Sep 23 20:15:15 dist-test-slave-vl24 krb5kdc[5984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194514, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:15.072477 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.056211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:60201 (local address 127.5.83.126:35485)
0923 20:15:15.056414 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.056420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.056732 (+ 312us) server_negotiation.cc:407] Connection header received
0923 20:15:15.058586 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.058594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.058715 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.058829 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:15.060778 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.061620 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.063164 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.063675 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.063772 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.066957 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:15.066962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:15.066965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:15.067003 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:15.069726 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.070481 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.070485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.070488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.070639 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.071020 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.071023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.071025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.071297 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:15.071409 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.072048 (+ 639us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.072231 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:15:15.073442 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.055608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:35485 (local address 127.5.83.67:60201)
0923 20:15:15.056495 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:15.056594 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:15.058319 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:15.059053 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:15.059061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:15.059787 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:15.060595 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.060622 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.061779 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.061784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.063017 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.063033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.063859 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.063864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.064045 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.064690 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:15.064714 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:15.066753 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:15.069849 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:15.069856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:15.069868 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:15.070339 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:15.070747 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:15.070750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:15.070752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:15.070922 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:15.071399 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:15.071404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:15.071714 (+ 310us) client_negotiation.cc:769] Sending connection context
0923 20:15:15.071936 (+ 222us) client_negotiation.cc:240] Negotiation successful
0923 20:15:15.072324 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":358,"threads_started":1}
I20260923 20:15:15.074993 6485 heartbeater.cc:347] Connected to a master server at 127.5.83.126:35485
I20260923 20:15:15.075318 6485 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:15.076148 6485 heartbeater.cc:511] Master 127.5.83.126:35485 requested a full tablet report, sending...
I20260923 20:15:15.077549 6028 ts_manager.cc:194] Registered new tserver with Master: 932e3e8fefb04e2788293c2e40d14d27 (127.5.83.67:36917)
I20260923 20:15:15.078390 6028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:60201
I20260923 20:15:15.088300 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:15:15.089373 6218 heartbeater.cc:503] Master 127.5.83.126:35485 was elected leader, sending a full tablet report...
I20260923 20:15:15.107882 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.095765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.5.83.126:35485)
0923 20:15:15.095947 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.095951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.096493 (+ 542us) server_negotiation.cc:407] Connection header received
0923 20:15:15.096720 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.096724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.096785 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.096874 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:15.098301 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.099091 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.100310 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.100777 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.100872 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.103176 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:15.103180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:15.103183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:15.103215 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:15.105777 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.106318 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.106321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.106324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.106441 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.106741 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.106744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.106746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.107025 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:15.107223 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.107478 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.107633 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:15:15.116611 6028 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:15:15.124115 6028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:15:15.152132 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.139124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:36917 (local address 127.0.0.1:56470)
0923 20:15:15.139905 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:15.139962 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:15.140190 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:15.143799 (+ 3609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:15.143806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:15.143834 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:15.144319 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.144337 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.146851 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.146857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.148958 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.148971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.150995 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.151002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.151196 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.151254 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 20:15:15.151358 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:15:15.151646 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":3,"thread_start_us":921,"threads_started":1}
I20260923 20:15:15.152325 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.139870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:46865 (local address 127.0.0.1:38336)
0923 20:15:15.143101 (+ 3231us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:15.143119 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:15.143303 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:15.144080 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:15.144085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:15.144114 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:15.144665 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.144682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.146532 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.146538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.149588 (+ 3050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.149602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.151431 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.151439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.151613 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.151644 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 20:15:15.151718 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 20:15:15.151859 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3001,"mutex_wait_us":60,"thread_start_us":2873,"threads_started":1}
I20260923 20:15:15.152838 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.140863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:39749 (local address 127.0.0.1:60524)
0923 20:15:15.143691 (+ 2828us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:15.143709 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:15.143927 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:15.144906 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:15.144912 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:15.144936 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:15.145440 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.145451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.147330 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.147340 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.149872 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.149882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.152014 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.152021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.152210 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.152250 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:15:15.152354 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:15:15.152485 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2700,"mutex_wait_us":1,"thread_start_us":2449,"threads_started":1}
I20260923 20:15:15.153191 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.139973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.5.83.66:46865)
0923 20:15:15.142520 (+ 2547us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.142531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.143242 (+ 711us) server_negotiation.cc:407] Connection header received
0923 20:15:15.143590 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.143599 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.143838 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.144668 (+ 830us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:15.144874 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.146388 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.149764 (+ 3376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.151241 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.151372 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.151591 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.152708 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.152877 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2355,"thread_start_us":2224,"threads_started":1}
I20260923 20:15:15.153532 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.141252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.5.83.65:39749)
0923 20:15:15.142012 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.142023 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.143811 (+ 1788us) server_negotiation.cc:407] Connection header received
0923 20:15:15.144215 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.144225 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.144456 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.144692 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:15.145603 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.147139 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.150061 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.151482 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.151594 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.151818 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.153064 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.153202 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":436,"threads_started":1}
I20260923 20:15:15.153573 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.139414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.5.83.67:36917)
0923 20:15:15.143079 (+ 3665us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.143087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.143127 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:15:15.143250 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.143261 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.143543 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.144326 (+ 783us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:15.144665 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.146702 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.149118 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.150798 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.151707 (+ 909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.151925 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.151997 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.152190 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3460,"thread_start_us":3289,"threads_started":1}
I20260923 20:15:15.157578 6153 tablet_service.cc:1467] Processing CreateTablet for tablet 108b9d373abc4e7b8779c7252533f6be (DEFAULT_TABLE table=test-table [id=d2c17fa35e7742bc8bb6af12ce2281eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:15.157630 6287 tablet_service.cc:1467] Processing CreateTablet for tablet 108b9d373abc4e7b8779c7252533f6be (DEFAULT_TABLE table=test-table [id=d2c17fa35e7742bc8bb6af12ce2281eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:15.157635 6420 tablet_service.cc:1467] Processing CreateTablet for tablet 108b9d373abc4e7b8779c7252533f6be (DEFAULT_TABLE table=test-table [id=d2c17fa35e7742bc8bb6af12ce2281eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:15.158962 6153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108b9d373abc4e7b8779c7252533f6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:15.159026 6420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108b9d373abc4e7b8779c7252533f6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:15.159027 6287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108b9d373abc4e7b8779c7252533f6be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:15.178614 6504 tablet_bootstrap.cc:492] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27: Bootstrap starting.
I20260923 20:15:15.179467 6503 tablet_bootstrap.cc:492] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3: Bootstrap starting.
I20260923 20:15:15.182006 6505 tablet_bootstrap.cc:492] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251: Bootstrap starting.
I20260923 20:15:15.183280 6503 tablet_bootstrap.cc:654] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:15.184469 6504 tablet_bootstrap.cc:654] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:15.184747 6503 log.cc:824] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:15.185694 6505 tablet_bootstrap.cc:654] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:15.185845 6504 log.cc:824] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:15.186966 6505 log.cc:824] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:15.188414 6503 tablet_bootstrap.cc:492] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3: No bootstrap required, opened a new log
I20260923 20:15:15.188750 6503 ts_tablet_manager.cc:1397] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260923 20:15:15.189174 6504 tablet_bootstrap.cc:492] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27: No bootstrap required, opened a new log
I20260923 20:15:15.189445 6504 ts_tablet_manager.cc:1397] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260923 20:15:15.190183 6505 tablet_bootstrap.cc:492] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251: No bootstrap required, opened a new log
I20260923 20:15:15.190444 6505 ts_tablet_manager.cc:1397] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:15:15.197595 6504 raft_consensus.cc:348] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932e3e8fefb04e2788293c2e40d14d27" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36917 } } peers { permanent_uuid: "6d6237dadd074ca6b080cbb1f1091251" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 39749 } } peers { permanent_uuid: "475dd05cb5d2491ebc76ed18091ceef3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46865 } }
I20260923 20:15:15.197981 6504 raft_consensus.cc:374] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:15.198071 6504 raft_consensus.cc:729] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932e3e8fefb04e2788293c2e40d14d27, State: Initialized, Role: FOLLOWER
I20260923 20:15:15.198722 6504 consensus_queue.cc:260] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932e3e8fefb04e2788293c2e40d14d27" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36917 } } peers { permanent_uuid: "6d6237dadd074ca6b080cbb1f1091251" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 39749 } } peers { permanent_uuid: "475dd05cb5d2491ebc76ed18091ceef3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46865 } }
I20260923 20:15:15.199605 6503 raft_consensus.cc:348] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932e3e8fefb04e2788293c2e40d14d27" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36917 } } peers { permanent_uuid: "6d6237dadd074ca6b080cbb1f1091251" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 39749 } } peers { permanent_uuid: "475dd05cb5d2491ebc76ed18091ceef3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46865 } }
I20260923 20:15:15.199957 6485 heartbeater.cc:503] Master 127.5.83.126:35485 was elected leader, sending a full tablet report...
I20260923 20:15:15.200006 6503 raft_consensus.cc:374] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:15.200129 6503 raft_consensus.cc:729] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 475dd05cb5d2491ebc76ed18091ceef3, State: Initialized, Role: FOLLOWER
I20260923 20:15:15.200698 6504 ts_tablet_manager.cc:1428] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27: Time spent starting tablet: real 0.011s user 0.005s sys 0.007s
I20260923 20:15:15.200834 6503 consensus_queue.cc:260] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932e3e8fefb04e2788293c2e40d14d27" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36917 } } peers { permanent_uuid: "6d6237dadd074ca6b080cbb1f1091251" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 39749 } } peers { permanent_uuid: "475dd05cb5d2491ebc76ed18091ceef3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46865 } }
I20260923 20:15:15.201086 6505 raft_consensus.cc:348] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932e3e8fefb04e2788293c2e40d14d27" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36917 } } peers { permanent_uuid: "6d6237dadd074ca6b080cbb1f1091251" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 39749 } } peers { permanent_uuid: "475dd05cb5d2491ebc76ed18091ceef3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46865 } }
I20260923 20:15:15.201535 6505 raft_consensus.cc:374] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:15.201647 6505 raft_consensus.cc:729] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6237dadd074ca6b080cbb1f1091251, State: Initialized, Role: FOLLOWER
I20260923 20:15:15.202189 6505 consensus_queue.cc:260] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932e3e8fefb04e2788293c2e40d14d27" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36917 } } peers { permanent_uuid: "6d6237dadd074ca6b080cbb1f1091251" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 39749 } } peers { permanent_uuid: "475dd05cb5d2491ebc76ed18091ceef3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46865 } }
I20260923 20:15:15.202636 6352 heartbeater.cc:503] Master 127.5.83.126:35485 was elected leader, sending a full tablet report...
I20260923 20:15:15.202750 6503 ts_tablet_manager.cc:1428] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3: Time spent starting tablet: real 0.014s user 0.002s sys 0.011s
I20260923 20:15:15.205154 6505 ts_tablet_manager.cc:1428] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
W20260923 20:15:15.297420 6353 tablet.cc:2369] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:15:15.303839 6486 tablet.cc:2369] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:15:15.307776 6219 tablet.cc:2369] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:15:15.395614 6509 raft_consensus.cc:482] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:15.395931 6509 raft_consensus.cc:504] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932e3e8fefb04e2788293c2e40d14d27" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36917 } } peers { permanent_uuid: "6d6237dadd074ca6b080cbb1f1091251" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 39749 } } peers { permanent_uuid: "475dd05cb5d2491ebc76ed18091ceef3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46865 } }
I20260923 20:15:15.397656 6510 raft_consensus.cc:482] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:15.397935 6510 raft_consensus.cc:504] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932e3e8fefb04e2788293c2e40d14d27" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36917 } } peers { permanent_uuid: "6d6237dadd074ca6b080cbb1f1091251" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 39749 } } peers { permanent_uuid: "475dd05cb5d2491ebc76ed18091ceef3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46865 } }
I20260923 20:15:15.398766 6509 leader_election.cc:288] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d6237dadd074ca6b080cbb1f1091251 (127.5.83.65:39749), 475dd05cb5d2491ebc76ed18091ceef3 (127.5.83.66:46865)
I20260923 20:15:15.399524 6510 leader_election.cc:288] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 932e3e8fefb04e2788293c2e40d14d27 (127.5.83.67:36917), 6d6237dadd074ca6b080cbb1f1091251 (127.5.83.65:39749)
I20260923 20:15:15.407534 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.398587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:39749 (local address 127.5.83.67:39301)
0923 20:15:15.399811 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:15.399829 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:15.399948 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:15.401068 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:15.401072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:15.401097 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:15.401301 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.401312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.402672 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.402677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.404618 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.404635 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.406659 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.406665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.406892 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.406924 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:15:15.407010 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 20:15:15.407134 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"mutex_wait_us":1,"thread_start_us":1368,"threads_started":1}
I20260923 20:15:15.408200 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.399157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:39301 (local address 127.5.83.65:39749)
0923 20:15:15.399373 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.399378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.399953 (+ 575us) server_negotiation.cc:407] Connection header received
0923 20:15:15.400743 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.400751 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.400840 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.400922 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:15.401472 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.402492 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.404809 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.406354 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.406476 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.406584 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.407819 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.407970 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:15:15.411207 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.399675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:34967 (local address 127.5.83.66:46865)
0923 20:15:15.399848 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.399855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.399876 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:15:15.399945 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.399949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.400020 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.400108 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:15.400869 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.402050 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.408807 (+ 6757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.410070 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.410206 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.410291 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.410832 (+ 541us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.410966 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:15:15.411893 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.398516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:46865 (local address 127.5.83.67:34967)
0923 20:15:15.399515 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:15.399537 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:15.399800 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:15.400214 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:15.400219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:15.400252 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:15.400696 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.400708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.402238 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.402243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.408611 (+ 6368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.408627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.410454 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.410459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.410653 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.410683 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:15:15.411416 (+ 733us) client_negotiation.cc:240] Negotiation successful
0923 20:15:15.411563 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":4071}
I20260923 20:15:15.412837 6307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108b9d373abc4e7b8779c7252533f6be" candidate_uuid: "932e3e8fefb04e2788293c2e40d14d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475dd05cb5d2491ebc76ed18091ceef3" is_pre_election: true
I20260923 20:15:15.413595 6307 raft_consensus.cc:2454] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 932e3e8fefb04e2788293c2e40d14d27 in term 0.
I20260923 20:15:15.413797 6173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108b9d373abc4e7b8779c7252533f6be" candidate_uuid: "932e3e8fefb04e2788293c2e40d14d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6237dadd074ca6b080cbb1f1091251" is_pre_election: true
I20260923 20:15:15.414376 6173 raft_consensus.cc:2454] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 932e3e8fefb04e2788293c2e40d14d27 in term 0.
I20260923 20:15:15.414587 6376 leader_election.cc:302] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 475dd05cb5d2491ebc76ed18091ceef3, 932e3e8fefb04e2788293c2e40d14d27; no voters:
I20260923 20:15:15.415015 6509 raft_consensus.cc:2781] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:15:15.415175 6509 raft_consensus.cc:482] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:15:15.415287 6509 raft_consensus.cc:3037] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:15.418598 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.400415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:59801 (local address 127.5.83.67:36917)
0923 20:15:15.400632 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.400640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.404997 (+ 4357us) server_negotiation.cc:407] Connection header received
0923 20:15:15.405236 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.405241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.405316 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.405370 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:15.406167 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.407239 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.416036 (+ 8797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.417397 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.417496 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.417607 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.418171 (+ 564us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.418301 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 20:15:15.418813 6509 raft_consensus.cc:504] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932e3e8fefb04e2788293c2e40d14d27" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36917 } } peers { permanent_uuid: "6d6237dadd074ca6b080cbb1f1091251" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 39749 } } peers { permanent_uuid: "475dd05cb5d2491ebc76ed18091ceef3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46865 } }
I20260923 20:15:15.419443 6513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.400384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:36917 (local address 127.5.83.66:59801)
0923 20:15:15.404915 (+ 4531us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:15.404932 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:15.405100 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:15.405526 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:15.405530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:15.405553 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:15.405998 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.406009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.407400 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.407405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.415851 (+ 8446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.415867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.417819 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.417825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.417993 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.418026 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:15:15.418895 (+ 869us) client_negotiation.cc:240] Negotiation successful
0923 20:15:15.419120 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4415,"thread_start_us":4218,"threads_started":1}
I20260923 20:15:15.419878 6173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108b9d373abc4e7b8779c7252533f6be" candidate_uuid: "932e3e8fefb04e2788293c2e40d14d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6237dadd074ca6b080cbb1f1091251"
I20260923 20:15:15.420156 6173 raft_consensus.cc:3037] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:15.420611 6509 leader_election.cc:288] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [CANDIDATE]: Term 1 election: Requested vote from peers 6d6237dadd074ca6b080cbb1f1091251 (127.5.83.65:39749), 475dd05cb5d2491ebc76ed18091ceef3 (127.5.83.66:46865)
I20260923 20:15:15.421336 6440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108b9d373abc4e7b8779c7252533f6be" candidate_uuid: "475dd05cb5d2491ebc76ed18091ceef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932e3e8fefb04e2788293c2e40d14d27" is_pre_election: true
I20260923 20:15:15.421945 6440 raft_consensus.cc:2379] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 475dd05cb5d2491ebc76ed18091ceef3 in current term 1: Already voted for candidate 932e3e8fefb04e2788293c2e40d14d27 in this term.
I20260923 20:15:15.423370 6307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108b9d373abc4e7b8779c7252533f6be" candidate_uuid: "932e3e8fefb04e2788293c2e40d14d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "475dd05cb5d2491ebc76ed18091ceef3"
I20260923 20:15:15.423576 6307 raft_consensus.cc:3037] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:15.423769 6173 raft_consensus.cc:2454] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 932e3e8fefb04e2788293c2e40d14d27 in term 1.
I20260923 20:15:15.424185 6373 leader_election.cc:302] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d6237dadd074ca6b080cbb1f1091251, 932e3e8fefb04e2788293c2e40d14d27; no voters:
I20260923 20:15:15.424575 6509 raft_consensus.cc:2781] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:15.425259 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.405062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:51971 (local address 127.5.83.65:39749)
0923 20:15:15.408407 (+ 3345us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.408414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.417787 (+ 9373us) server_negotiation.cc:407] Connection header received
0923 20:15:15.418279 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.418283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.418376 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.418434 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:15.419133 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.420288 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.422651 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.424052 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.424162 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.424270 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.424857 (+ 587us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.424981 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3255,"thread_start_us":7589,"threads_started":1}
I20260923 20:15:15.425870 6509 raft_consensus.cc:686] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [term 1 LEADER]: Becoming Leader. State: Replica: 932e3e8fefb04e2788293c2e40d14d27, State: Running, Role: LEADER
I20260923 20:15:15.425972 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.406426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:39749 (local address 127.5.83.66:51971)
0923 20:15:15.417686 (+ 11260us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:15.417705 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:15.418108 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:15.418640 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:15.418645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:15.418670 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:15.418970 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.418980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.420440 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.420445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.422474 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:15.422487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.424444 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.424453 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:15.424675 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.424705 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:15:15.425378 (+ 673us) client_negotiation.cc:240] Negotiation successful
0923 20:15:15.425522 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11130,"thread_start_us":11369,"threads_started":1}
I20260923 20:15:15.426419 6509 consensus_queue.cc:237] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932e3e8fefb04e2788293c2e40d14d27" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36917 } } peers { permanent_uuid: "6d6237dadd074ca6b080cbb1f1091251" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 39749 } } peers { permanent_uuid: "475dd05cb5d2491ebc76ed18091ceef3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46865 } }
I20260923 20:15:15.426563 6173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108b9d373abc4e7b8779c7252533f6be" candidate_uuid: "475dd05cb5d2491ebc76ed18091ceef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d6237dadd074ca6b080cbb1f1091251" is_pre_election: true
I20260923 20:15:15.426841 6173 raft_consensus.cc:2379] T 108b9d373abc4e7b8779c7252533f6be P 6d6237dadd074ca6b080cbb1f1091251 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 475dd05cb5d2491ebc76ed18091ceef3 in current term 1: Already voted for candidate 932e3e8fefb04e2788293c2e40d14d27 in this term.
I20260923 20:15:15.427248 6307 raft_consensus.cc:2454] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 932e3e8fefb04e2788293c2e40d14d27 in term 1.
I20260923 20:15:15.427482 6240 leader_election.cc:302] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 475dd05cb5d2491ebc76ed18091ceef3; no voters: 6d6237dadd074ca6b080cbb1f1091251, 932e3e8fefb04e2788293c2e40d14d27
I20260923 20:15:15.427878 6510 raft_consensus.cc:2726] T 108b9d373abc4e7b8779c7252533f6be P 475dd05cb5d2491ebc76ed18091ceef3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 20:15:15.432446 6027 catalog_manager.cc:5495] T 108b9d373abc4e7b8779c7252533f6be P 932e3e8fefb04e2788293c2e40d14d27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 932e3e8fefb04e2788293c2e40d14d27 (127.5.83.67). New cstate: current_term: 1 leader_uuid: "932e3e8fefb04e2788293c2e40d14d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932e3e8fefb04e2788293c2e40d14d27" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d6237dadd074ca6b080cbb1f1091251" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 39749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "475dd05cb5d2491ebc76ed18091ceef3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46865 } health_report { overall_health: UNKNOWN } } }
Sep 23 20:15:15 dist-test-slave-vl24 krb5kdc[5984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:15.454409 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.439792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.5.83.65:39749)
0923 20:15:15.440004 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.440012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.440423 (+ 411us) server_negotiation.cc:407] Connection header received
0923 20:15:15.440607 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.440611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.440688 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.440775 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:15.442301 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.443003 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.444175 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.444662 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.444754 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.448931 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:15.448947 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:15.448963 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:15.449004 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:15.451994 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.452718 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.452725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.452728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.452851 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.453229 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.453232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.453234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.453546 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:15.453766 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.453992 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.454152 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 20:15:15 dist-test-slave-vl24 krb5kdc[5984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194515, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:15:15 dist-test-slave-vl24 krb5kdc[5984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194515, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:15.502166 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.487499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.5.83.65:39749)
0923 20:15:15.487687 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.487695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.488174 (+ 479us) server_negotiation.cc:407] Connection header received
0923 20:15:15.488314 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.488320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.488414 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.488508 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:15.490019 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.490761 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.492229 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.492819 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.492966 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.496872 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:15.496878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:15.496881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:15.496917 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:15.499476 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.500109 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.500115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.500119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.500283 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.500761 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.500768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.500771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.501025 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:15.501295 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.501559 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.501768 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 20:15:15 dist-test-slave-vl24 krb5kdc[5984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:15:15 dist-test-slave-vl24 krb5kdc[5984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:15.548348 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.533562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.5.83.65:39749)
0923 20:15:15.533801 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.533808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.534053 (+ 245us) server_negotiation.cc:407] Connection header received
0923 20:15:15.534279 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.534284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.534350 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.534458 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:15.535915 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.536725 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.537974 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.538478 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.538617 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.542666 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:15.542670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:15.542674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:15.542706 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:15.545711 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.546438 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.546446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.546449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.546585 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.546947 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.546953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.546956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.547229 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:15.547504 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.547801 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.548019 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260923 20:15:15.550658 6133 tablet_service.cc:2908] Rejecting scan request for tablet 108b9d373abc4e7b8779c7252533f6be: Uninitialized: clean time has not yet been initialized
I20260923 20:15:15.554169 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 6090
I20260923 20:15:15.567411 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.555674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.5.83.65:39749)
0923 20:15:15.555874 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.555882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.556193 (+ 311us) server_negotiation.cc:407] Connection header received
0923 20:15:15.556384 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.556389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.556494 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.556626 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:15.557782 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.558460 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.559637 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.560049 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.560186 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.562434 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:15.562440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:15.562445 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:15.562483 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:15.564860 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.565468 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.565476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.565479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.565602 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.565952 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.565958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.565961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.566304 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:15.566564 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.566872 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.567088 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 20:15:15.568023 6213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:15.659054 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 6090
I20260923 20:15:15.675721 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 6224
Sep 23 20:15:15 dist-test-slave-vl24 krb5kdc[5984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:15:15.689642 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.676574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38350 (local address 127.5.83.66:46865)
0923 20:15:15.676765 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.676770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.676788 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:15:15.676844 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.676847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.676908 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.676991 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:15.678097 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.678846 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.680082 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.680511 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.680749 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.684493 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:15.684506 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:15.684521 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:15.684599 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:15.687138 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.687834 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.687842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.687844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.687964 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.688282 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.688285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.688287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.688809 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:15.689027 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.689234 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.689390 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:15:15.690299 6347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:15.780035 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 6224
I20260923 20:15:15.796706 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 6357
Sep 23 20:15:15 dist-test-slave-vl24 krb5kdc[5984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:15:15.811420 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.797400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.5.83.67:36917)
0923 20:15:15.797585 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:15.797593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:15.797676 (+ 83us) server_negotiation.cc:407] Connection header received
0923 20:15:15.797901 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:15.797905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:15.797967 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:15.798072 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:15.799281 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.800010 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.801389 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:15.801852 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:15.801983 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:15.806136 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:15.806150 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:15.806167 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:15.806198 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:15.808943 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.809586 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.809594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.809596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.809727 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:15.810044 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:15.810047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:15.810049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:15.810428 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:15.810654 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:15.810869 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 20:15:15.811031 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:15:15.812052 6480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:15.903380 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:15.902673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:39749 (local address 127.5.83.67:45467)
0923 20:15:15.902998 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:15.903167 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 20:15:15.903481 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 6357
I20260923 20:15:15.920109 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 5997
I20260923 20:15:15.921307 6059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:16.029724 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 5997
2026-09-23T20:15:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4327 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:15:16 dist-test-slave-vl24 krb5kdc[6535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:15:16 dist-test-slave-vl24 krb5kdc[6535](info): set up 2 sockets
Sep 23 20:15:16 dist-test-slave-vl24 krb5kdc[6535](info): commencing operation
krb5kdc: starting...
W20260923 20:15:17.117832 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:15:17 dist-test-slave-vl24 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:15:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:15:17.305975 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:38837
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:38837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:17.624568 6548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:17.624850 6548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:17.625006 6548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:17.632366 6548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:17.632475 6548 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 20:15:17.632576 6548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:17.632700 6548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:17.632773 6548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:17.632855 6548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:15:17.642989 6548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:38837
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:38837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:17.644259 6548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:17.645628 6548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:17.656917 6554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:17.657119 6556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:17.657416 6553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:17.658061 6548 server_base.cc:1034] running on GCE node
I20260923 20:15:17.658855 6548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:17.660344 6548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:17.661614 6548 hybrid_clock.cc:648] HybridClock initialized: now 1790194517661539 us; error 46 us; skew 500 ppm
I20260923 20:15:17.662078 6548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:17 dist-test-slave-vl24 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194517, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:17.665393 6548 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:15:17.666868 6548 webserver.cc:460] Webserver started at http://127.5.83.126:32965/ using document root <none> and password file <none>
I20260923 20:15:17.667634 6548 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:17.667773 6548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:17.668128 6548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:17.671422 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "6597b5c21f674c00bf9df83771af8c1d"
format_stamp: "Formatted at 2026-09-23 20:15:17 on dist-test-slave-vl24"
server_key: "7192ee148a8f8077765937566b0bc1f5"
server_key_iv: "c2ccc4accc2d40a5c41c67d53cf642ee"
server_key_version: "encryptionkey@0"
I20260923 20:15:17.672444 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "6597b5c21f674c00bf9df83771af8c1d"
format_stamp: "Formatted at 2026-09-23 20:15:17 on dist-test-slave-vl24"
server_key: "7192ee148a8f8077765937566b0bc1f5"
server_key_iv: "c2ccc4accc2d40a5c41c67d53cf642ee"
server_key_version: "encryptionkey@0"
I20260923 20:15:17.678030 6548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:15:17.681875 6563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:17.683324 6548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:15:17.683527 6548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "6597b5c21f674c00bf9df83771af8c1d"
format_stamp: "Formatted at 2026-09-23 20:15:17 on dist-test-slave-vl24"
server_key: "7192ee148a8f8077765937566b0bc1f5"
server_key_iv: "c2ccc4accc2d40a5c41c67d53cf642ee"
server_key_version: "encryptionkey@0"
I20260923 20:15:17.683763 6548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:17.701788 6548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:17.706147 6548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:17.706426 6548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:17.727890 6548 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:38837
I20260923 20:15:17.727916 6624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:38837 every 8 connection(s)
I20260923 20:15:17.729313 6548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:17.733074 6625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:17.733728 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 6548
I20260923 20:15:17.733935 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:17.734601 5453 external_mini_cluster.cc:1428] Setting key 5bb8c43ea0a5aa5d5c731d7c4121ebdf
I20260923 20:15:17.747241 6625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d: Bootstrap starting.
Sep 23 20:15:17 dist-test-slave-vl24 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:17.750353 6625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:17.751534 6625 log.cc:824] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:17.755019 6625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d: No bootstrap required, opened a new log
I20260923 20:15:17.759268 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:17.736796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.5.83.126:38837)
0923 20:15:17.737670 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:17.737686 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:17.737735 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:15:17.739694 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:17.739722 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:17.740254 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:17.740715 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:17.742171 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:17.744198 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:17.745612 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:17.746604 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:17.746790 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:17.751169 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:17.751187 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:17.751209 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:17.751266 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:17.755062 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:17.755811 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:17.755819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:17.755828 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:17.756001 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:17.756408 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:17.756417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:17.756420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:17.757056 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:17.757383 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:17.757823 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 20:15:17.758259 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":374,"threads_started":1}
I20260923 20:15:17.763734 6625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6597b5c21f674c00bf9df83771af8c1d" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38837 } }
I20260923 20:15:17.764082 6625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:17.764178 6625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6597b5c21f674c00bf9df83771af8c1d, State: Initialized, Role: FOLLOWER
I20260923 20:15:17.764820 6625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6597b5c21f674c00bf9df83771af8c1d" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38837 } }
I20260923 20:15:17.765050 6625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:17.765172 6625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:17.765331 6625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:17.767968 6625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6597b5c21f674c00bf9df83771af8c1d" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38837 } }
I20260923 20:15:17.768499 6625 leader_election.cc:302] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6597b5c21f674c00bf9df83771af8c1d; no voters:
I20260923 20:15:17.769151 6625 leader_election.cc:288] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:17.769285 6630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:17.769959 6630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [term 1 LEADER]: Becoming Leader. State: Replica: 6597b5c21f674c00bf9df83771af8c1d, State: Running, Role: LEADER
I20260923 20:15:17.770469 6630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6597b5c21f674c00bf9df83771af8c1d" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38837 } }
I20260923 20:15:17.771384 6625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:17.773640 6631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6597b5c21f674c00bf9df83771af8c1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6597b5c21f674c00bf9df83771af8c1d" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38837 } } }
I20260923 20:15:17.773615 6632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6597b5c21f674c00bf9df83771af8c1d. Latest consensus state: current_term: 1 leader_uuid: "6597b5c21f674c00bf9df83771af8c1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6597b5c21f674c00bf9df83771af8c1d" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38837 } } }
I20260923 20:15:17.774029 6632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:17.774029 6631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:17.775935 6636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:17.781559 6636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:17.790875 6636 catalog_manager.cc:1347] Generated new cluster ID: c50323728879408090cf341ac094db68
I20260923 20:15:17.790999 6636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:17.802691 6636 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:17.804283 6636 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:17.810809 6636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d: Generated new TSK 0
I20260923 20:15:17.811668 6636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:15:17.856204 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:38837
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:18.180409 6651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:18.180714 6651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:18.180907 6651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:18.188395 6651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:18.188566 6651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:18.188742 6651 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:15:18.188872 6651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:15:18.198724 6651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.126:38837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:18.200012 6651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:18.201431 6651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:18.215068 6659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:18.215032 6657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:18.215116 6656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:18.215525 6651 server_base.cc:1034] running on GCE node
I20260923 20:15:18.216212 6651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:18.217156 6651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:18.218389 6651 hybrid_clock.cc:648] HybridClock initialized: now 1790194518218340 us; error 33 us; skew 500 ppm
I20260923 20:15:18.218678 6651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:18 dist-test-slave-vl24 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194518, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:18.221613 6651 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:15:18.223031 6651 webserver.cc:460] Webserver started at http://127.5.83.65:37817/ using document root <none> and password file <none>
I20260923 20:15:18.223732 6651 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:18.223863 6651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:18.224188 6651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:18.227099 6651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "6fccb863bdf84b24949b4322353b00e7"
format_stamp: "Formatted at 2026-09-23 20:15:18 on dist-test-slave-vl24"
server_key: "772b85be130b6ab0c1e38f005101679d"
server_key_iv: "ab600be2495d861efb88589203398d04"
server_key_version: "encryptionkey@0"
I20260923 20:15:18.227861 6651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "6fccb863bdf84b24949b4322353b00e7"
format_stamp: "Formatted at 2026-09-23 20:15:18 on dist-test-slave-vl24"
server_key: "772b85be130b6ab0c1e38f005101679d"
server_key_iv: "ab600be2495d861efb88589203398d04"
server_key_version: "encryptionkey@0"
I20260923 20:15:18.232882 6651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260923 20:15:18.236202 6666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:18.237495 6651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:18.237675 6651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "6fccb863bdf84b24949b4322353b00e7"
format_stamp: "Formatted at 2026-09-23 20:15:18 on dist-test-slave-vl24"
server_key: "772b85be130b6ab0c1e38f005101679d"
server_key_iv: "ab600be2495d861efb88589203398d04"
server_key_version: "encryptionkey@0"
I20260923 20:15:18.237879 6651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:18.251242 6651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:18.255182 6651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:18.255458 6651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:18.258178 6651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:18.258299 6651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:18.258471 6651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:18.258549 6651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:15:18 dist-test-slave-vl24 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194518, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:18.283493 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:18.258883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:50143 (local address 127.5.83.126:38837)
0923 20:15:18.259363 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:18.259369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:18.260517 (+ 1148us) server_negotiation.cc:407] Connection header received
0923 20:15:18.262846 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:18.262852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:18.262969 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:18.263077 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:18.265914 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.266848 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.270144 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.270632 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.270789 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:18.276273 (+ 5484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:18.276284 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:18.276292 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:18.276353 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:18.280014 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:18.280983 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:18.280987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:18.280989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:18.281104 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:18.281758 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:18.281763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:18.281767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:18.282053 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:18.282218 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:18.282934 (+ 716us) server_negotiation.cc:299] Negotiation successful
0923 20:15:18.283148 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341}
I20260923 20:15:18.284653 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:18.259482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:38837 (local address 127.5.83.65:50143)
0923 20:15:18.260333 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:18.260388 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:18.262566 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:18.263482 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:18.263502 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:18.264504 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:18.265680 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:18.265713 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.267031 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.267040 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:18.269941 (+ 2901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:18.269975 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.270929 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.270939 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:18.271313 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:18.272093 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:18.272132 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:18.275955 (+ 3823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:18.280159 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:18.280168 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:18.280186 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:18.280807 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:18.281283 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:18.281290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:18.281296 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:18.281607 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:18.282189 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:18.282198 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:18.282556 (+ 358us) client_negotiation.cc:769] Sending connection context
0923 20:15:18.282842 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 20:15:18.283394 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":414,"threads_started":1}
I20260923 20:15:18.305632 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:18.290408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:43333 (local address 127.5.83.126:38837)
0923 20:15:18.290558 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:18.290562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:18.291189 (+ 627us) server_negotiation.cc:407] Connection header received
0923 20:15:18.291394 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:18.291399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:18.291471 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:18.291565 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:18.292445 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.293423 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.295819 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.296236 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.296376 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:18.299315 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:18.299323 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:18.299329 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:18.299375 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:18.302545 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:18.303426 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:18.303429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:18.303432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:18.303547 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:18.304022 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:18.304028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:18.304033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:18.304365 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:18.304529 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:18.304994 (+ 465us) server_negotiation.cc:299] Negotiation successful
0923 20:15:18.305270 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 20:15:18.305756 6734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:18.290451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:38837 (local address 127.5.83.65:43333)
0923 20:15:18.291087 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:18.291105 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:18.291268 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:18.291585 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:18.291590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:18.292028 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:18.292310 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:18.292318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.293597 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.293607 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:18.295663 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:18.295681 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.296422 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.296428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:18.296652 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:18.297429 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:18.297455 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:18.299075 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:18.302683 (+ 3608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:18.302688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:18.302691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:18.303308 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:18.303665 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:18.303670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:18.303677 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:18.303841 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:18.304487 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:18.304491 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:18.304712 (+ 221us) client_negotiation.cc:769] Sending connection context
0923 20:15:18.304898 (+ 186us) client_negotiation.cc:240] Negotiation successful
0923 20:15:18.305290 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":393,"threads_started":1}
I20260923 20:15:18.316834 6651 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:33471
I20260923 20:15:18.316864 6785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:33471 every 8 connection(s)
I20260923 20:15:18.318120 6651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:15:18.320104 6786 heartbeater.cc:347] Connected to a master server at 127.5.83.126:38837
I20260923 20:15:18.320343 6786 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:18.321060 6786 heartbeater.cc:511] Master 127.5.83.126:38837 requested a full tablet report, sending...
I20260923 20:15:18.322983 6580 ts_manager.cc:194] Registered new tserver with Master: 6fccb863bdf84b24949b4322353b00e7 (127.5.83.65:33471)
I20260923 20:15:18.324440 6580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:50143
I20260923 20:15:18.328438 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 6651
I20260923 20:15:18.328665 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:15:18.329300 5453 external_mini_cluster.cc:1428] Setting key 5d01af943921409aebc9a52a7b2b4db7
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:15:18.368206 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:38837
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:18.686160 6790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:18.686508 6790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:18.686693 6790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:18.693929 6790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:18.694044 6790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:18.694211 6790 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:15:18.694331 6790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:15:18.704350 6790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.126:38837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:18.705746 6790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:18.707079 6790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:18.720129 6796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:18.720225 6795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:18.720131 6798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:18.721091 6790 server_base.cc:1034] running on GCE node
I20260923 20:15:18.721634 6790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:18.722541 6790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:18.723757 6790 hybrid_clock.cc:648] HybridClock initialized: now 1790194518723705 us; error 26 us; skew 500 ppm
I20260923 20:15:18.724009 6790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:18 dist-test-slave-vl24 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194518, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:18.727134 6790 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:15:18.728736 6790 webserver.cc:460] Webserver started at http://127.5.83.66:41013/ using document root <none> and password file <none>
I20260923 20:15:18.729418 6790 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:18.729504 6790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:18.729844 6790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:18.732743 6790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "9ad8e4ce4f3a497cbff2228e1630992f"
format_stamp: "Formatted at 2026-09-23 20:15:18 on dist-test-slave-vl24"
server_key: "a46d3c16a6399bf638f66dd70d0df22b"
server_key_iv: "0285abfdf76dcc55783542345ebc50c4"
server_key_version: "encryptionkey@0"
I20260923 20:15:18.733526 6790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "9ad8e4ce4f3a497cbff2228e1630992f"
format_stamp: "Formatted at 2026-09-23 20:15:18 on dist-test-slave-vl24"
server_key: "a46d3c16a6399bf638f66dd70d0df22b"
server_key_iv: "0285abfdf76dcc55783542345ebc50c4"
server_key_version: "encryptionkey@0"
I20260923 20:15:18.738785 6790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:15:18.742434 6805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:18.743741 6790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:15:18.743906 6790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "9ad8e4ce4f3a497cbff2228e1630992f"
format_stamp: "Formatted at 2026-09-23 20:15:18 on dist-test-slave-vl24"
server_key: "a46d3c16a6399bf638f66dd70d0df22b"
server_key_iv: "0285abfdf76dcc55783542345ebc50c4"
server_key_version: "encryptionkey@0"
I20260923 20:15:18.744128 6790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:18.788259 6790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:18.792294 6790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:18.792594 6790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:18.795109 6790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:18.795220 6790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:18.795372 6790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:18.795450 6790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:15:18 dist-test-slave-vl24 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194518, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:18.819845 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:18.795916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:57001 (local address 127.5.83.126:38837)
0923 20:15:18.796102 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:18.796108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:18.797749 (+ 1641us) server_negotiation.cc:407] Connection header received
0923 20:15:18.800891 (+ 3142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:18.800899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:18.801022 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:18.801133 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:18.803411 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.804755 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.806587 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.807185 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.807321 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:18.812546 (+ 5225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:18.812553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:18.812557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:18.812605 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:18.816310 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:18.817329 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:18.817338 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:18.817344 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:18.817508 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:18.818134 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:18.818139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:18.818143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:18.818429 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:18.818597 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:18.819294 (+ 697us) server_negotiation.cc:299] Negotiation successful
0923 20:15:18.819516 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:15:18.821097 6814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:18.796599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:38837 (local address 127.5.83.66:57001)
0923 20:15:18.797556 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:18.797605 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:18.800498 (+ 2893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:18.801448 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:18.801463 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:18.802302 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:18.803178 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:18.803211 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.804953 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.804965 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:18.806420 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:18.806431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.807429 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.807436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:18.807659 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:18.808486 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:18.808550 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:18.812289 (+ 3739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:18.816468 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:18.816477 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:18.816499 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:18.817153 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:18.817651 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:18.817659 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:18.817665 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:18.818005 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:18.818566 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:18.818574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:18.818925 (+ 351us) client_negotiation.cc:769] Sending connection context
0923 20:15:18.819180 (+ 255us) client_negotiation.cc:240] Negotiation successful
0923 20:15:18.819701 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":397,"threads_started":1}
I20260923 20:15:18.844578 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:18.827855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:40813 (local address 127.5.83.126:38837)
0923 20:15:18.828049 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:18.828056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:18.828933 (+ 877us) server_negotiation.cc:407] Connection header received
0923 20:15:18.829128 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:18.829133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:18.829224 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:18.829353 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:18.830339 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.831657 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.834072 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.834866 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.834990 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:18.837235 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:18.837245 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:18.837250 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:18.837289 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:18.841222 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:18.842064 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:18.842072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:18.842078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:18.842249 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:18.842733 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:18.842737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:18.842740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:18.843172 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:18.843342 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:18.843905 (+ 563us) server_negotiation.cc:299] Negotiation successful
0923 20:15:18.844167 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:15:18.844748 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:18.827963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:38837 (local address 127.5.83.66:40813)
0923 20:15:18.828767 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:18.828791 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:18.828993 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:18.829396 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:18.829401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:18.829909 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:18.830183 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:18.830199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.831809 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.831815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:18.833900 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:18.833920 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:18.835074 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:18.835079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:18.835187 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:18.835849 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:18.835894 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:18.837032 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:18.841375 (+ 4343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:18.841383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:18.841390 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:18.841912 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:18.842428 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:18.842436 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:18.842443 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:18.842609 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:18.843331 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:18.843339 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:18.843584 (+ 245us) client_negotiation.cc:769] Sending connection context
0923 20:15:18.843725 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 20:15:18.844135 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":421,"threads_started":1}
I20260923 20:15:18.844821 6790 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:43679
I20260923 20:15:18.844868 6924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:43679 every 8 connection(s)
I20260923 20:15:18.846146 6790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:15:18.848202 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 6790
I20260923 20:15:18.848424 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:15:18.848961 5453 external_mini_cluster.cc:1428] Setting key 8e47163c8c13b1dc12dc47fd2727d801
I20260923 20:15:18.848795 6925 heartbeater.cc:347] Connected to a master server at 127.5.83.126:38837
I20260923 20:15:18.849783 6925 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:18.850517 6925 heartbeater.cc:511] Master 127.5.83.126:38837 requested a full tablet report, sending...
I20260923 20:15:18.851852 6580 ts_manager.cc:194] Registered new tserver with Master: 9ad8e4ce4f3a497cbff2228e1630992f (127.5.83.66:43679)
I20260923 20:15:18.852747 6580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:57001
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:15:18.888469 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:38837
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:19.217765 6929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:19.218089 6929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:19.218257 6929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:19.225835 6929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:19.225998 6929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:19.226146 6929 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:15:19.226280 6929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:15:19.236353 6929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.126:38837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:19.237897 6929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:19.239300 6929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:19.253082 6935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:19.253170 6937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:19.253113 6934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:19.254554 6929 server_base.cc:1034] running on GCE node
I20260923 20:15:19.255328 6929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:19.256335 6929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:19.257583 6929 hybrid_clock.cc:648] HybridClock initialized: now 1790194519257508 us; error 41 us; skew 500 ppm
I20260923 20:15:19.257948 6929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:19 dist-test-slave-vl24 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194519, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:19.267426 6929 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:15:19.269157 6929 webserver.cc:460] Webserver started at http://127.5.83.67:40949/ using document root <none> and password file <none>
I20260923 20:15:19.269876 6929 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:19.269971 6929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:19.270263 6929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:19.273445 6929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "120ee2de42f24cf0b5cd4d39d141412e"
format_stamp: "Formatted at 2026-09-23 20:15:19 on dist-test-slave-vl24"
server_key: "d182ae323e9171bc027a95be9612b92f"
server_key_iv: "bb5e3abcb87e0435fcc3759c044506bd"
server_key_version: "encryptionkey@0"
I20260923 20:15:19.274247 6929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "120ee2de42f24cf0b5cd4d39d141412e"
format_stamp: "Formatted at 2026-09-23 20:15:19 on dist-test-slave-vl24"
server_key: "d182ae323e9171bc027a95be9612b92f"
server_key_iv: "bb5e3abcb87e0435fcc3759c044506bd"
server_key_version: "encryptionkey@0"
I20260923 20:15:19.279901 6929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 20:15:19.283468 6944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:19.284920 6929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:15:19.285115 6929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "120ee2de42f24cf0b5cd4d39d141412e"
format_stamp: "Formatted at 2026-09-23 20:15:19 on dist-test-slave-vl24"
server_key: "d182ae323e9171bc027a95be9612b92f"
server_key_iv: "bb5e3abcb87e0435fcc3759c044506bd"
server_key_version: "encryptionkey@0"
I20260923 20:15:19.285363 6929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:19.313962 6929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:19.318809 6929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:19.319109 6929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:19.322175 6929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:19.322295 6929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:19.322463 6929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:19.322546 6929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:19.327071 6786 heartbeater.cc:503] Master 127.5.83.126:38837 was elected leader, sending a full tablet report...
Sep 23 20:15:19 dist-test-slave-vl24 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194519, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:19.348274 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.322947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:37951 (local address 127.5.83.126:38837)
0923 20:15:19.323174 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.323183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.324718 (+ 1535us) server_negotiation.cc:407] Connection header received
0923 20:15:19.326710 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.326715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.326804 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.326949 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:19.329770 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.330499 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.334160 (+ 3661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.334762 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.334906 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.340913 (+ 6007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:19.340922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:19.340927 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:19.340998 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:19.344676 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:19.345807 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:19.345813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:19.345815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:19.345990 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:19.346462 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:19.346467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:19.346471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:19.346765 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:19.346923 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.347665 (+ 742us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.347915 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:15:19.349771 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.323569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:38837 (local address 127.5.83.67:37951)
0923 20:15:19.324509 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:19.324595 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:19.326424 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:19.327263 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:19.327285 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:19.328347 (+ 1062us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:19.329558 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.329594 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.330685 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.330694 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.333949 (+ 3255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.333972 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.335270 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.335280 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.335644 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.336477 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:19.336520 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:19.340510 (+ 3990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:19.344853 (+ 4343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:19.344863 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:19.344893 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:19.345654 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:19.346127 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:19.346132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:19.346136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:19.346337 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:19.346897 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:19.346905 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:19.347292 (+ 387us) client_negotiation.cc:769] Sending connection context
0923 20:15:19.347602 (+ 310us) client_negotiation.cc:240] Negotiation successful
0923 20:15:19.348154 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":426,"threads_started":1}
I20260923 20:15:19.377012 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.358779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:39985 (local address 127.5.83.126:38837)
0923 20:15:19.359024 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.359033 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.360345 (+ 1312us) server_negotiation.cc:407] Connection header received
0923 20:15:19.360775 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.360779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.360860 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.360942 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:19.362728 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.363402 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.366329 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.366933 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.367093 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.369821 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:19.369832 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:19.369835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:19.369866 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:19.373691 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:19.374613 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:19.374621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:19.374628 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:19.374838 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:19.375340 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:19.375347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:19.375353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:19.375709 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:19.375869 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.376342 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.376604 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:15:19.377116 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.358725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:38837 (local address 127.5.83.67:39985)
0923 20:15:19.360146 (+ 1421us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:19.360218 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:19.360454 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:19.361074 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:19.361082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:19.361918 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:19.362462 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.362480 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.363576 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.363585 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.366149 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.366173 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.367173 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.367188 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.367516 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.368646 (+ 1130us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:19.368671 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:19.369638 (+ 967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:19.373832 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:19.373838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:19.373843 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:19.374408 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:19.374964 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:19.374970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:19.374974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:19.375172 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:19.375834 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:19.375839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:19.376033 (+ 194us) client_negotiation.cc:769] Sending connection context
0923 20:15:19.376221 (+ 188us) client_negotiation.cc:240] Negotiation successful
0923 20:15:19.376604 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"thread_start_us":912,"threads_started":1}
I20260923 20:15:19.386845 6929 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:45167
I20260923 20:15:19.386868 7063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:45167 every 8 connection(s)
I20260923 20:15:19.388329 6929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:15:19.390779 7064 heartbeater.cc:347] Connected to a master server at 127.5.83.126:38837
I20260923 20:15:19.390975 7064 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:19.391577 7064 heartbeater.cc:511] Master 127.5.83.126:38837 requested a full tablet report, sending...
I20260923 20:15:19.392899 6580 ts_manager.cc:194] Registered new tserver with Master: 120ee2de42f24cf0b5cd4d39d141412e (127.5.83.67:45167)
I20260923 20:15:19.393680 6580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:37951
I20260923 20:15:19.396847 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 6929
I20260923 20:15:19.396991 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:15:19.397389 5453 external_mini_cluster.cc:1428] Setting key fba8841814bb5b962850bf94bc389305
I20260923 20:15:19.400820 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:15:19 dist-test-slave-vl24 krb5kdc[6535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:15:19 dist-test-slave-vl24 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:19.447150 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.433467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.5.83.126:38837)
0923 20:15:19.433728 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.433733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.433834 (+ 101us) server_negotiation.cc:407] Connection header received
0923 20:15:19.433986 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.433991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.434057 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.434146 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:19.435471 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.436194 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.437458 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.438043 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.438180 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.442330 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:19.442335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:19.442358 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:19.442394 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:19.445016 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:19.445570 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:19.445574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:19.445576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:19.445712 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:19.446017 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:19.446020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:19.446022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:19.446270 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:19.446436 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.446714 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.446889 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 20:15:19.451471 6580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:15:19.458454 6580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:15:19.480644 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.470212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:45167 (local address 127.0.0.1:42950)
0923 20:15:19.471269 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:19.471311 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:19.471471 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:19.472665 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:19.472673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:19.472707 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:19.473278 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.473293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.475292 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.475303 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.477767 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.477780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.479515 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.479521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.479725 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.479783 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 20:15:19.479961 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 20:15:19.480150 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":21,"thread_start_us":691,"threads_started":1}
I20260923 20:15:19.481374 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.470471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.5.83.67:45167)
0923 20:15:19.472054 (+ 1583us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.472063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.472107 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:15:19.472207 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.472214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.472430 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.472666 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:19.473543 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.475129 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.477985 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.479311 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.479462 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.479728 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.480842 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.481012 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1334,"thread_start_us":2534,"threads_started":1}
I20260923 20:15:19.481436 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.470503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:43679 (local address 127.0.0.1:41290)
0923 20:15:19.473394 (+ 2891us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:19.473414 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:19.473524 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:19.474370 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:19.474376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:19.474400 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:19.474722 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.474734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.476420 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.476425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.478648 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.478662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.480663 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.480670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.480839 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.480872 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:15:19.480960 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 20:15:19.481094 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2759,"thread_start_us":2597,"threads_started":1}
I20260923 20:15:19.482159 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.471716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.5.83.66:43679)
0923 20:15:19.472203 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.472210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.473640 (+ 1430us) server_negotiation.cc:407] Connection header received
0923 20:15:19.473742 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.473751 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.473989 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.474186 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:19.474886 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.476073 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.478824 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.480183 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.480298 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.480480 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.481723 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.481902 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":842,"threads_started":1}
I20260923 20:15:19.482636 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.471233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:33471 (local address 127.0.0.1:41768)
0923 20:15:19.474053 (+ 2820us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:19.474075 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:19.474191 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:19.475218 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:19.475223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:19.475251 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:19.475748 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.475759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.477705 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.477711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.479925 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.479939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.481876 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.481881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.482042 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.482072 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:15:19.482164 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 20:15:19.482300 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2683,"mutex_wait_us":3,"thread_start_us":2745,"threads_started":1}
I20260923 20:15:19.482688 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.472326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.5.83.65:33471)
0923 20:15:19.474679 (+ 2353us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.474686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.474712 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:15:19.474781 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.474785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.475019 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.475212 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:19.475949 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.477547 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.480127 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.481597 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.481717 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.481910 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.482231 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.482404 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2219,"thread_start_us":2086,"threads_started":1}
I20260923 20:15:19.485526 6994 tablet_service.cc:1467] Processing CreateTablet for tablet b797875fecd545d5a5b25b8ab675723c (DEFAULT_TABLE table=test-table [id=7756491c2f1542738d69ecf5cbcaee7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:19.485677 6855 tablet_service.cc:1467] Processing CreateTablet for tablet b797875fecd545d5a5b25b8ab675723c (DEFAULT_TABLE table=test-table [id=7756491c2f1542738d69ecf5cbcaee7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:19.486274 6716 tablet_service.cc:1467] Processing CreateTablet for tablet b797875fecd545d5a5b25b8ab675723c (DEFAULT_TABLE table=test-table [id=7756491c2f1542738d69ecf5cbcaee7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:19.486997 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b797875fecd545d5a5b25b8ab675723c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:19.487061 6855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b797875fecd545d5a5b25b8ab675723c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:19.487525 6716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b797875fecd545d5a5b25b8ab675723c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:19.499110 7083 tablet_bootstrap.cc:492] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e: Bootstrap starting.
I20260923 20:15:19.501637 7085 tablet_bootstrap.cc:492] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7: Bootstrap starting.
I20260923 20:15:19.504052 7083 tablet_bootstrap.cc:654] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:19.505201 7084 tablet_bootstrap.cc:492] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f: Bootstrap starting.
I20260923 20:15:19.505262 7085 tablet_bootstrap.cc:654] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:19.505394 7083 log.cc:824] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:19.506464 7085 log.cc:824] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:19.508522 7083 tablet_bootstrap.cc:492] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e: No bootstrap required, opened a new log
I20260923 20:15:19.508841 7083 ts_tablet_manager.cc:1397] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260923 20:15:19.508971 7084 tablet_bootstrap.cc:654] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:19.509498 7085 tablet_bootstrap.cc:492] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7: No bootstrap required, opened a new log
I20260923 20:15:19.509763 7085 ts_tablet_manager.cc:1397] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260923 20:15:19.510051 7084 log.cc:824] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:19.513060 7084 tablet_bootstrap.cc:492] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f: No bootstrap required, opened a new log
I20260923 20:15:19.513322 7084 ts_tablet_manager.cc:1397] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260923 20:15:19.520346 7084 raft_consensus.cc:348] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } }
I20260923 20:15:19.520311 7085 raft_consensus.cc:348] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } }
I20260923 20:15:19.520311 7083 raft_consensus.cc:348] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } }
I20260923 20:15:19.520787 7084 raft_consensus.cc:374] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:19.520787 7085 raft_consensus.cc:374] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:19.520787 7083 raft_consensus.cc:374] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:19.520884 7083 raft_consensus.cc:729] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120ee2de42f24cf0b5cd4d39d141412e, State: Initialized, Role: FOLLOWER
I20260923 20:15:19.520884 7085 raft_consensus.cc:729] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fccb863bdf84b24949b4322353b00e7, State: Initialized, Role: FOLLOWER
I20260923 20:15:19.520879 7084 raft_consensus.cc:729] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad8e4ce4f3a497cbff2228e1630992f, State: Initialized, Role: FOLLOWER
I20260923 20:15:19.521528 7084 consensus_queue.cc:260] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } }
I20260923 20:15:19.521541 7085 consensus_queue.cc:260] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } }
I20260923 20:15:19.521543 7083 consensus_queue.cc:260] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } }
I20260923 20:15:19.523252 7064 heartbeater.cc:503] Master 127.5.83.126:38837 was elected leader, sending a full tablet report...
I20260923 20:15:19.524386 7083 ts_tablet_manager.cc:1428] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260923 20:15:19.525192 7085 ts_tablet_manager.cc:1428] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7: Time spent starting tablet: real 0.015s user 0.006s sys 0.009s
I20260923 20:15:19.525607 6925 heartbeater.cc:503] Master 127.5.83.126:38837 was elected leader, sending a full tablet report...
I20260923 20:15:19.526126 7084 ts_tablet_manager.cc:1428] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260923 20:15:19.539570 7090 raft_consensus.cc:482] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:19.539911 7090 raft_consensus.cc:504] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } }
I20260923 20:15:19.541973 7090 leader_election.cc:288] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 120ee2de42f24cf0b5cd4d39d141412e (127.5.83.67:45167), 9ad8e4ce4f3a497cbff2228e1630992f (127.5.83.66:43679)
I20260923 20:15:19.549018 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.542374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:54567 (local address 127.5.83.67:45167)
0923 20:15:19.542616 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.542625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.543404 (+ 779us) server_negotiation.cc:407] Connection header received
0923 20:15:19.543621 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.543626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.543686 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.543786 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:19.544440 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.545256 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.547134 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.547993 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.548174 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.548293 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.548603 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.548735 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:15:19.549319 7092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.542209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:45167 (local address 127.5.83.65:54567)
0923 20:15:19.543290 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:19.543306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:19.543479 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:19.543798 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:19.543802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:19.543824 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:19.544293 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.544306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.545394 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.545400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.546938 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.546979 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.548235 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.548241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.548401 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.548425 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:15:19.548540 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 20:15:19.548721 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":691,"threads_started":1}
I20260923 20:15:19.549485 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.542403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:43351 (local address 127.5.83.66:43679)
0923 20:15:19.542608 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.542617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.543491 (+ 874us) server_negotiation.cc:407] Connection header received
0923 20:15:19.543689 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.543695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.543765 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.543867 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:19.544655 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.545367 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.547196 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.548363 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.548549 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.548673 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.549096 (+ 423us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.549226 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:15:19.549715 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.542381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:43679 (local address 127.5.83.65:43351)
0923 20:15:19.543385 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:19.543400 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:19.543547 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:19.543889 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:19.543894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:19.543932 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:19.544449 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.544458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.545511 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.545516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.547039 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.547051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.548777 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.548792 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.548929 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.548954 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:15:19.549073 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 20:15:19.549203 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":1,"thread_start_us":574,"threads_started":1}
I20260923 20:15:19.550477 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b797875fecd545d5a5b25b8ab675723c" candidate_uuid: "6fccb863bdf84b24949b4322353b00e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120ee2de42f24cf0b5cd4d39d141412e" is_pre_election: true
I20260923 20:15:19.550587 6879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b797875fecd545d5a5b25b8ab675723c" candidate_uuid: "6fccb863bdf84b24949b4322353b00e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" is_pre_election: true
I20260923 20:15:19.551007 7019 raft_consensus.cc:2454] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fccb863bdf84b24949b4322353b00e7 in term 0.
I20260923 20:15:19.551077 6879 raft_consensus.cc:2454] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fccb863bdf84b24949b4322353b00e7 in term 0.
I20260923 20:15:19.551841 6670 leader_election.cc:302] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 120ee2de42f24cf0b5cd4d39d141412e, 6fccb863bdf84b24949b4322353b00e7, 9ad8e4ce4f3a497cbff2228e1630992f; no voters:
I20260923 20:15:19.552249 7090 raft_consensus.cc:2781] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:15:19.552402 7090 raft_consensus.cc:482] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:15:19.552563 7090 raft_consensus.cc:3037] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:19.554996 7090 raft_consensus.cc:504] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } }
I20260923 20:15:19.555630 7090 leader_election.cc:288] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [CANDIDATE]: Term 1 election: Requested vote from peers 120ee2de42f24cf0b5cd4d39d141412e (127.5.83.67:45167), 9ad8e4ce4f3a497cbff2228e1630992f (127.5.83.66:43679)
I20260923 20:15:19.556006 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b797875fecd545d5a5b25b8ab675723c" candidate_uuid: "6fccb863bdf84b24949b4322353b00e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120ee2de42f24cf0b5cd4d39d141412e"
I20260923 20:15:19.556126 6879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b797875fecd545d5a5b25b8ab675723c" candidate_uuid: "6fccb863bdf84b24949b4322353b00e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad8e4ce4f3a497cbff2228e1630992f"
I20260923 20:15:19.556221 7019 raft_consensus.cc:3037] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:19.556344 6879 raft_consensus.cc:3037] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:19.559894 6879 raft_consensus.cc:2454] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fccb863bdf84b24949b4322353b00e7 in term 1.
I20260923 20:15:19.559907 7019 raft_consensus.cc:2454] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fccb863bdf84b24949b4322353b00e7 in term 1.
I20260923 20:15:19.560364 6669 leader_election.cc:302] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 120ee2de42f24cf0b5cd4d39d141412e, 6fccb863bdf84b24949b4322353b00e7, 9ad8e4ce4f3a497cbff2228e1630992f; no voters:
I20260923 20:15:19.560777 7090 raft_consensus.cc:2781] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:19.561232 7090 raft_consensus.cc:686] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 1 LEADER]: Becoming Leader. State: Replica: 6fccb863bdf84b24949b4322353b00e7, State: Running, Role: LEADER
I20260923 20:15:19.561764 7090 consensus_queue.cc:237] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } }
I20260923 20:15:19.566907 6580 catalog_manager.cc:5495] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fccb863bdf84b24949b4322353b00e7 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "6fccb863bdf84b24949b4322353b00e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } health_report { overall_health: HEALTHY } } }
W20260923 20:15:19.570528 6787 tablet.cc:2369] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:15:19.595114 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.589338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41770 (local address 127.5.83.65:33471)
0923 20:15:19.589550 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.589557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.589713 (+ 156us) server_negotiation.cc:407] Connection header received
0923 20:15:19.590140 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.590146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.590263 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.590413 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:15:19.590978 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.591713 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.593140 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.593527 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.593623 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.594028 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:15:19.594558 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:15:19.594672 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.594838 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.594901 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 20:15:19.598819 6926 tablet.cc:2369] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:15:19.603932 7019 raft_consensus.cc:1261] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e [term 1 FOLLOWER]: Refusing update from remote peer 6fccb863bdf84b24949b4322353b00e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:19.605026 7094 consensus_queue.cc:1035] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:19.605974 6879 raft_consensus.cc:1261] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f [term 1 FOLLOWER]: Refusing update from remote peer 6fccb863bdf84b24949b4322353b00e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:19.606745 7094 consensus_queue.cc:1035] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:19.616911 7097 mvcc.cc:205] Tried to move back new op lower bound from 7332636752286990336 to 7332636752128999424. Current Snapshot: MvccSnapshot[applied={T|T < 7332636752286990336}]
I20260923 20:15:19.639554 6579 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:19.639945 6579 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:19.644112 6579 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d: Sending DeleteTablet for 3 replicas of tablet b797875fecd545d5a5b25b8ab675723c
I20260923 20:15:19.645535 6855 tablet_service.cc:1514] Processing DeleteTablet for tablet b797875fecd545d5a5b25b8ab675723c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:19 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:41290
I20260923 20:15:19.645519 6994 tablet_service.cc:1514] Processing DeleteTablet for tablet b797875fecd545d5a5b25b8ab675723c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:19 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:42950
I20260923 20:15:19.646363 6716 tablet_service.cc:1514] Processing DeleteTablet for tablet b797875fecd545d5a5b25b8ab675723c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:19 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:41768
I20260923 20:15:19.648450 6579 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:15:19.649397 6579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:15:19.658808 6715 tablet_service.cc:1467] Processing CreateTablet for tablet 604e0a297fc8449eb691e87be27e2ac3 (DEFAULT_TABLE table=test-table [id=e241918788574003b490da9fa306358d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:19.659024 7108 tablet_replica.cc:330] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f: stopping tablet replica
I20260923 20:15:19.659054 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 604e0a297fc8449eb691e87be27e2ac3 (DEFAULT_TABLE table=test-table [id=e241918788574003b490da9fa306358d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:19.659518 6715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604e0a297fc8449eb691e87be27e2ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:19.659570 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604e0a297fc8449eb691e87be27e2ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:19.659626 7108 raft_consensus.cc:2229] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:19.660151 7108 raft_consensus.cc:2258] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:19.661350 7106 tablet_replica.cc:330] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e: stopping tablet replica
I20260923 20:15:19.661773 7106 raft_consensus.cc:2229] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:19.662259 7106 raft_consensus.cc:2258] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:19.664701 7107 tablet_replica.cc:330] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7: stopping tablet replica
I20260923 20:15:19.665016 6855 tablet_service.cc:1467] Processing CreateTablet for tablet 604e0a297fc8449eb691e87be27e2ac3 (DEFAULT_TABLE table=test-table [id=e241918788574003b490da9fa306358d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:19.665083 7107 raft_consensus.cc:2229] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:15:19.665444 6855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604e0a297fc8449eb691e87be27e2ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:19.665932 7107 raft_consensus.cc:2258] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:19.666119 7108 ts_tablet_manager.cc:1905] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:19.667981 7107 ts_tablet_manager.cc:1905] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:19.671325 7106 ts_tablet_manager.cc:1905] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:19.676633 7083 tablet_bootstrap.cc:492] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e: Bootstrap starting.
I20260923 20:15:19.677480 7085 tablet_bootstrap.cc:492] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7: Bootstrap starting.
I20260923 20:15:19.678354 7084 tablet_bootstrap.cc:492] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f: Bootstrap starting.
I20260923 20:15:19.678450 7083 tablet_bootstrap.cc:654] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:19.679215 7085 tablet_bootstrap.cc:654] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:19.680008 7084 tablet_bootstrap.cc:654] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:19.685725 7108 ts_tablet_manager.cc:1918] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:15:19.685902 7108 log.cc:1196] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/b797875fecd545d5a5b25b8ab675723c
I20260923 20:15:19.690002 7085 tablet_bootstrap.cc:492] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7: No bootstrap required, opened a new log
I20260923 20:15:19.690395 7085 ts_tablet_manager.cc:1397] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260923 20:15:19.691656 7084 tablet_bootstrap.cc:492] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f: No bootstrap required, opened a new log
I20260923 20:15:19.691681 7108 ts_tablet_manager.cc:1939] T b797875fecd545d5a5b25b8ab675723c P 9ad8e4ce4f3a497cbff2228e1630992f: Deleting consensus metadata
I20260923 20:15:19.691823 7084 ts_tablet_manager.cc:1397] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.007s
I20260923 20:15:19.691576 7085 raft_consensus.cc:348] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.692108 7085 raft_consensus.cc:374] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:19.692269 7085 raft_consensus.cc:729] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fccb863bdf84b24949b4322353b00e7, State: Initialized, Role: FOLLOWER
I20260923 20:15:19.692708 7085 consensus_queue.cc:260] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.692840 7084 raft_consensus.cc:348] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.693542 7085 ts_tablet_manager.cc:1428] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 20:15:19.694152 7107 ts_tablet_manager.cc:1918] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:15:19.694377 7107 log.cc:1196] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/b797875fecd545d5a5b25b8ab675723c
I20260923 20:15:19.695158 7107 ts_tablet_manager.cc:1939] T b797875fecd545d5a5b25b8ab675723c P 6fccb863bdf84b24949b4322353b00e7: Deleting consensus metadata
I20260923 20:15:19.695545 7106 ts_tablet_manager.cc:1918] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:15:19.695737 7106 log.cc:1196] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/b797875fecd545d5a5b25b8ab675723c
I20260923 20:15:19.696275 7106 ts_tablet_manager.cc:1939] T b797875fecd545d5a5b25b8ab675723c P 120ee2de42f24cf0b5cd4d39d141412e: Deleting consensus metadata
I20260923 20:15:19.697479 6564 catalog_manager.cc:4841] TS 6fccb863bdf84b24949b4322353b00e7 (127.5.83.65:33471): tablet b797875fecd545d5a5b25b8ab675723c (table test-table [id=7756491c2f1542738d69ecf5cbcaee7c]) successfully deleted
I20260923 20:15:19.697592 6566 catalog_manager.cc:4841] TS 9ad8e4ce4f3a497cbff2228e1630992f (127.5.83.66:43679): tablet b797875fecd545d5a5b25b8ab675723c (table test-table [id=7756491c2f1542738d69ecf5cbcaee7c]) successfully deleted
I20260923 20:15:19.693079 7084 raft_consensus.cc:374] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:19.698128 7084 raft_consensus.cc:729] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad8e4ce4f3a497cbff2228e1630992f, State: Initialized, Role: FOLLOWER
I20260923 20:15:19.698489 7084 consensus_queue.cc:260] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.698721 6567 catalog_manager.cc:4841] TS 120ee2de42f24cf0b5cd4d39d141412e (127.5.83.67:45167): tablet b797875fecd545d5a5b25b8ab675723c (table test-table [id=7756491c2f1542738d69ecf5cbcaee7c]) successfully deleted
I20260923 20:15:19.699322 7084 ts_tablet_manager.cc:1428] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260923 20:15:19.700522 7083 tablet_bootstrap.cc:492] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e: No bootstrap required, opened a new log
I20260923 20:15:19.700711 7083 ts_tablet_manager.cc:1397] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.004s
I20260923 20:15:19.701489 7083 raft_consensus.cc:348] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.701723 7083 raft_consensus.cc:374] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:19.701817 7083 raft_consensus.cc:729] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120ee2de42f24cf0b5cd4d39d141412e, State: Initialized, Role: FOLLOWER
I20260923 20:15:19.702059 7083 consensus_queue.cc:260] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.702762 7083 ts_tablet_manager.cc:1428] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:15:19.731941 7089 raft_consensus.cc:482] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:19.732182 7089 raft_consensus.cc:504] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.734141 7089 leader_election.cc:288] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fccb863bdf84b24949b4322353b00e7 (127.5.83.65:33471), 120ee2de42f24cf0b5cd4d39d141412e (127.5.83.67:45167)
I20260923 20:15:19.735214 7090 raft_consensus.cc:482] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:19.735409 7090 raft_consensus.cc:504] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.736053 7090 leader_election.cc:288] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ad8e4ce4f3a497cbff2228e1630992f (127.5.83.66:43679), 120ee2de42f24cf0b5cd4d39d141412e (127.5.83.67:45167)
I20260923 20:15:19.736866 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604e0a297fc8449eb691e87be27e2ac3" candidate_uuid: "6fccb863bdf84b24949b4322353b00e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120ee2de42f24cf0b5cd4d39d141412e" is_pre_election: true
I20260923 20:15:19.737106 7019 raft_consensus.cc:2454] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fccb863bdf84b24949b4322353b00e7 in term 0.
I20260923 20:15:19.737743 6670 leader_election.cc:302] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 120ee2de42f24cf0b5cd4d39d141412e, 6fccb863bdf84b24949b4322353b00e7; no voters:
I20260923 20:15:19.738106 7090 raft_consensus.cc:2781] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:15:19.738286 7090 raft_consensus.cc:482] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:15:19.738400 7090 raft_consensus.cc:3037] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:19.743681 7090 raft_consensus.cc:504] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.745287 6879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604e0a297fc8449eb691e87be27e2ac3" candidate_uuid: "6fccb863bdf84b24949b4322353b00e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" is_pre_election: true
I20260923 20:15:19.745553 6879 raft_consensus.cc:2454] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fccb863bdf84b24949b4322353b00e7 in term 0.
I20260923 20:15:19.745971 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604e0a297fc8449eb691e87be27e2ac3" candidate_uuid: "6fccb863bdf84b24949b4322353b00e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120ee2de42f24cf0b5cd4d39d141412e"
I20260923 20:15:19.746191 7019 raft_consensus.cc:3037] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:19.746169 6879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604e0a297fc8449eb691e87be27e2ac3" candidate_uuid: "6fccb863bdf84b24949b4322353b00e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad8e4ce4f3a497cbff2228e1630992f"
I20260923 20:15:19.746433 6879 raft_consensus.cc:3037] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:19.749599 6879 raft_consensus.cc:2454] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fccb863bdf84b24949b4322353b00e7 in term 1.
I20260923 20:15:19.749596 7019 raft_consensus.cc:2454] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fccb863bdf84b24949b4322353b00e7 in term 1.
I20260923 20:15:19.750217 6670 leader_election.cc:302] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 120ee2de42f24cf0b5cd4d39d141412e, 6fccb863bdf84b24949b4322353b00e7; no voters:
I20260923 20:15:19.750564 7094 raft_consensus.cc:2781] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:19.753521 7090 leader_election.cc:288] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [CANDIDATE]: Term 1 election: Requested vote from peers 9ad8e4ce4f3a497cbff2228e1630992f (127.5.83.66:43679), 120ee2de42f24cf0b5cd4d39d141412e (127.5.83.67:45167)
I20260923 20:15:19.757210 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.734887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:39749 (local address 127.5.83.65:33471)
0923 20:15:19.735059 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.735066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.745224 (+ 10158us) server_negotiation.cc:407] Connection header received
0923 20:15:19.745515 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.745521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.745605 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.745668 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:19.746581 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.747745 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.751138 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.752608 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.752727 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.752834 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.756656 (+ 3822us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.756841 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:15:19.757947 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.734470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:33471 (local address 127.5.83.66:39749)
0923 20:15:19.744993 (+ 10523us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:19.745014 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:19.745347 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:19.745878 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:19.745883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:19.745910 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:19.746315 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.746329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.747941 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.747946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.750444 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.750459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.753037 (+ 2578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.753043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.753234 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.753281 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 20:15:19.757465 (+ 4184us) client_negotiation.cc:240] Negotiation successful
0923 20:15:19.757624 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10373,"thread_start_us":10377,"threads_started":1}
I20260923 20:15:19.758049 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.734928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:40451 (local address 127.5.83.67:45167)
0923 20:15:19.735089 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.735095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.750673 (+ 15578us) server_negotiation.cc:407] Connection header received
0923 20:15:19.750935 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.750942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.751015 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.751084 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:19.751929 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.753156 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.755595 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.756878 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.756980 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.757082 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.757643 (+ 561us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.757795 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:15:19.758836 6741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604e0a297fc8449eb691e87be27e2ac3" candidate_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fccb863bdf84b24949b4322353b00e7" is_pre_election: true
I20260923 20:15:19.759552 7112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.734581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:45167 (local address 127.5.83.66:40451)
0923 20:15:19.750585 (+ 16004us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:19.750604 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:19.750787 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:19.751316 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:19.751321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:19.751348 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:19.751724 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.751735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.753335 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.753343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.755417 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.755431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.757281 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.757287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.757465 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.757498 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:15:19.758267 (+ 769us) client_negotiation.cc:240] Negotiation successful
0923 20:15:19.758408 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15861,"thread_start_us":10117,"threads_started":1}
I20260923 20:15:19.760232 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604e0a297fc8449eb691e87be27e2ac3" candidate_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120ee2de42f24cf0b5cd4d39d141412e" is_pre_election: true
I20260923 20:15:19.760650 7019 raft_consensus.cc:2379] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ad8e4ce4f3a497cbff2228e1630992f in current term 1: Already voted for candidate 6fccb863bdf84b24949b4322353b00e7 in this term.
I20260923 20:15:19.761827 7094 raft_consensus.cc:686] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 1 LEADER]: Becoming Leader. State: Replica: 6fccb863bdf84b24949b4322353b00e7, State: Running, Role: LEADER
I20260923 20:15:19.762192 7094 consensus_queue.cc:237] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.765457 6806 leader_election.cc:302] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ad8e4ce4f3a497cbff2228e1630992f; no voters: 120ee2de42f24cf0b5cd4d39d141412e, 6fccb863bdf84b24949b4322353b00e7
I20260923 20:15:19.765929 7089 raft_consensus.cc:2726] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 20:15:19.766506 6579 catalog_manager.cc:5495] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fccb863bdf84b24949b4322353b00e7 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "6fccb863bdf84b24949b4322353b00e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } health_report { overall_health: UNKNOWN } } }
Sep 23 20:15:19 dist-test-slave-vl24 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194517, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:19.822439 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.807056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.5.83.126:38837)
0923 20:15:19.807289 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.807298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.807953 (+ 655us) server_negotiation.cc:407] Connection header received
0923 20:15:19.808131 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.808135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.808200 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.808281 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:19.809878 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.810609 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.812451 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.812923 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.813015 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.816871 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:19.816876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:19.816879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:19.816906 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:19.819552 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:19.820269 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:19.820277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:19.820280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:19.820422 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:19.820873 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:19.820878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:19.820884 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:19.821149 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:19.821304 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.821688 (+ 384us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.822018 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":75}
I20260923 20:15:19.822459 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.807065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:38837 (local address 127.0.0.1:47114)
0923 20:15:19.807832 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:19.807853 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:19.808016 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:19.808352 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:19.808357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:19.809292 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:19.809723 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.809737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.810732 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.810738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.812308 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.812323 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.813110 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.813117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.813286 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.813969 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:19.813993 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:19.816674 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:19.819682 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:19.819709 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:19.819718 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:19.820155 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:19.820578 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:19.820584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:19.820587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:19.820767 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:19.821281 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:19.821289 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:19.821444 (+ 155us) client_negotiation.cc:769] Sending connection context
0923 20:15:19.821597 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 20:15:19.821888 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":447,"threads_started":1}
I20260923 20:15:19.828022 6579 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:47114:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 20:15:19.829146 6579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:15:19.836172 6716 tablet_service.cc:1467] Processing CreateTablet for tablet 470ae0e2eb5c448f8ede48b83600d3dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c12bea3c2464919848eacda925af2e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:15:19.836397 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 470ae0e2eb5c448f8ede48b83600d3dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c12bea3c2464919848eacda925af2e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:15:19.836850 6716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470ae0e2eb5c448f8ede48b83600d3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:19.837045 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470ae0e2eb5c448f8ede48b83600d3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:19.841683 6855 tablet_service.cc:1467] Processing CreateTablet for tablet 470ae0e2eb5c448f8ede48b83600d3dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c12bea3c2464919848eacda925af2e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:15:19.842219 6855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470ae0e2eb5c448f8ede48b83600d3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:19.843017 7083 tablet_bootstrap.cc:492] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e: Bootstrap starting.
I20260923 20:15:19.844664 7083 tablet_bootstrap.cc:654] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:19.853425 7084 tablet_bootstrap.cc:492] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f: Bootstrap starting.
I20260923 20:15:19.855259 7084 tablet_bootstrap.cc:654] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:19.857823 7084 tablet_bootstrap.cc:492] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f: No bootstrap required, opened a new log
I20260923 20:15:19.857982 7084 ts_tablet_manager.cc:1397] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260923 20:15:19.858174 7083 tablet_bootstrap.cc:492] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e: No bootstrap required, opened a new log
I20260923 20:15:19.858357 7083 ts_tablet_manager.cc:1397] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20260923 20:15:19.858697 7084 raft_consensus.cc:348] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.858934 7084 raft_consensus.cc:374] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:19.859011 7084 raft_consensus.cc:729] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad8e4ce4f3a497cbff2228e1630992f, State: Initialized, Role: FOLLOWER
I20260923 20:15:19.859012 7083 raft_consensus.cc:348] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.859279 7083 raft_consensus.cc:374] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:19.859361 7083 raft_consensus.cc:729] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120ee2de42f24cf0b5cd4d39d141412e, State: Initialized, Role: FOLLOWER
I20260923 20:15:19.859287 7084 consensus_queue.cc:260] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.859596 7083 consensus_queue.cc:260] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.859786 7089 tablet_replica.cc:439] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } }
I20260923 20:15:19.860039 7089 tablet_replica.cc:442] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:19.860014 7091 tablet_replica.cc:439] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } }
I20260923 20:15:19.860405 7091 tablet_replica.cc:442] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:19.862084 7083 ts_tablet_manager.cc:1428] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260923 20:15:19.862246 7084 ts_tablet_manager.cc:1428] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260923 20:15:19.864871 7085 tablet_bootstrap.cc:492] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: Bootstrap starting.
I20260923 20:15:19.866828 7085 tablet_bootstrap.cc:654] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:19.869163 7085 tablet_bootstrap.cc:492] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: No bootstrap required, opened a new log
I20260923 20:15:19.869360 7085 ts_tablet_manager.cc:1397] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260923 20:15:19.870119 7085 raft_consensus.cc:348] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.870422 7085 raft_consensus.cc:374] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:19.870527 7085 raft_consensus.cc:729] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fccb863bdf84b24949b4322353b00e7, State: Initialized, Role: FOLLOWER
I20260923 20:15:19.870818 7085 consensus_queue.cc:260] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.871411 7085 ts_tablet_manager.cc:1428] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260923 20:15:19.871238 7094 tablet_replica.cc:439] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } }
I20260923 20:15:19.872213 7094 tablet_replica.cc:442] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 20:15:19.891393 7065 tablet.cc:2369] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:15:19.940346 7094 raft_consensus.cc:482] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:19.940629 7094 raft_consensus.cc:504] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.941417 7094 leader_election.cc:288] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ad8e4ce4f3a497cbff2228e1630992f (127.5.83.66:43679), 120ee2de42f24cf0b5cd4d39d141412e (127.5.83.67:45167)
I20260923 20:15:19.942042 6879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470ae0e2eb5c448f8ede48b83600d3dd" candidate_uuid: "6fccb863bdf84b24949b4322353b00e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" is_pre_election: true
I20260923 20:15:19.942029 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470ae0e2eb5c448f8ede48b83600d3dd" candidate_uuid: "6fccb863bdf84b24949b4322353b00e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120ee2de42f24cf0b5cd4d39d141412e" is_pre_election: true
I20260923 20:15:19.942327 7019 raft_consensus.cc:2454] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fccb863bdf84b24949b4322353b00e7 in term 0.
I20260923 20:15:19.942342 6879 raft_consensus.cc:2454] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fccb863bdf84b24949b4322353b00e7 in term 0.
I20260923 20:15:19.942767 6670 leader_election.cc:302] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 120ee2de42f24cf0b5cd4d39d141412e, 6fccb863bdf84b24949b4322353b00e7; no voters:
I20260923 20:15:19.943045 7094 raft_consensus.cc:2781] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:15:19.943156 7094 raft_consensus.cc:482] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:15:19.943329 7094 raft_consensus.cc:3037] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:19.945658 7094 raft_consensus.cc:504] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.946709 6879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470ae0e2eb5c448f8ede48b83600d3dd" candidate_uuid: "6fccb863bdf84b24949b4322353b00e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad8e4ce4f3a497cbff2228e1630992f"
I20260923 20:15:19.946764 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470ae0e2eb5c448f8ede48b83600d3dd" candidate_uuid: "6fccb863bdf84b24949b4322353b00e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120ee2de42f24cf0b5cd4d39d141412e"
I20260923 20:15:19.946954 6879 raft_consensus.cc:3037] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:19.947118 7019 raft_consensus.cc:3037] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:19.947249 7094 leader_election.cc:288] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [CANDIDATE]: Term 1 election: Requested vote from peers 9ad8e4ce4f3a497cbff2228e1630992f (127.5.83.66:43679), 120ee2de42f24cf0b5cd4d39d141412e (127.5.83.67:45167)
I20260923 20:15:19.949330 7019 raft_consensus.cc:2454] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fccb863bdf84b24949b4322353b00e7 in term 1.
I20260923 20:15:19.949774 6670 leader_election.cc:302] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 120ee2de42f24cf0b5cd4d39d141412e, 6fccb863bdf84b24949b4322353b00e7; no voters:
I20260923 20:15:19.950058 7094 raft_consensus.cc:2781] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:19.950054 6879 raft_consensus.cc:2454] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fccb863bdf84b24949b4322353b00e7 in term 1.
I20260923 20:15:19.950273 7094 raft_consensus.cc:686] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [term 1 LEADER]: Becoming Leader. State: Replica: 6fccb863bdf84b24949b4322353b00e7, State: Running, Role: LEADER
I20260923 20:15:19.950546 7094 consensus_queue.cc:237] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } }
I20260923 20:15:19.952718 7114 tablet_replica.cc:439] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: TxnStatusTablet state changed. Reason: New leader 6fccb863bdf84b24949b4322353b00e7. Latest consensus state: current_term: 1 leader_uuid: "6fccb863bdf84b24949b4322353b00e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } }
I20260923 20:15:19.953215 7114 tablet_replica.cc:442] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:15:19.954079 7126 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 20:15:19.954411 7126 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 20:15:19.954396 6580 catalog_manager.cc:5495] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fccb863bdf84b24949b4322353b00e7 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "6fccb863bdf84b24949b4322353b00e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } health_report { overall_health: UNKNOWN } } }
I20260923 20:15:19.985214 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.979426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:33471 (local address 127.0.0.1:41776)
0923 20:15:19.979675 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:19.979696 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:19.979852 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:19.980291 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:19.980294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:19.980304 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:15:19.980455 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.980461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.981942 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.981946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.982931 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:19.982942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.983959 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.983963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:19.984161 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.984226 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:15:19.984715 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:15:19.984731 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 20:15:19.984796 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 20:15:19.984948 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 20:15:19.985206 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:19.979383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41776 (local address 127.5.83.65:33471)
0923 20:15:19.979587 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:19.979593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:19.979793 (+ 200us) server_negotiation.cc:407] Connection header received
0923 20:15:19.980028 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:19.980035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:19.980153 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:19.980254 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:15:19.980609 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.981810 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.983103 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:19.983803 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:19.983949 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:19.984444 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:15:19.984613 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:15:19.984715 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:19.984823 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 20:15:19.984904 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:15:19.989789 6879 raft_consensus.cc:1261] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f [term 1 FOLLOWER]: Refusing update from remote peer 6fccb863bdf84b24949b4322353b00e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:19.989789 7019 raft_consensus.cc:1261] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e [term 1 FOLLOWER]: Refusing update from remote peer 6fccb863bdf84b24949b4322353b00e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:19.990515 7114 consensus_queue.cc:1035] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:19.990864 7094 consensus_queue.cc:1035] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:19.994436 7089 tablet_replica.cc:439] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f: TxnStatusTablet state changed. Reason: New leader 6fccb863bdf84b24949b4322353b00e7. Latest consensus state: current_term: 1 leader_uuid: "6fccb863bdf84b24949b4322353b00e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } }
I20260923 20:15:19.994830 7089 tablet_replica.cc:442] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:19.997486 7114 tablet_replica.cc:439] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fccb863bdf84b24949b4322353b00e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } }
I20260923 20:15:19.997843 7114 tablet_replica.cc:442] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:15:20.001760 7089 tablet_replica.cc:439] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fccb863bdf84b24949b4322353b00e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } }
I20260923 20:15:20.002100 7089 tablet_replica.cc:442] T 470ae0e2eb5c448f8ede48b83600d3dd P 9ad8e4ce4f3a497cbff2228e1630992f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:20.010614 7091 tablet_replica.cc:439] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e: TxnStatusTablet state changed. Reason: New leader 6fccb863bdf84b24949b4322353b00e7. Latest consensus state: current_term: 1 leader_uuid: "6fccb863bdf84b24949b4322353b00e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } }
I20260923 20:15:20.010905 7091 tablet_replica.cc:442] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:20.012033 7094 tablet_replica.cc:439] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fccb863bdf84b24949b4322353b00e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } }
I20260923 20:15:20.012383 7094 tablet_replica.cc:442] T 470ae0e2eb5c448f8ede48b83600d3dd P 6fccb863bdf84b24949b4322353b00e7: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:15:20.013974 7098 mvcc.cc:205] Tried to move back new op lower bound from 7332636753871212544 to 7332636753719857152. Current Snapshot: MvccSnapshot[applied={T|T < 7332636753871212544}]
I20260923 20:15:20.015324 7091 tablet_replica.cc:439] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fccb863bdf84b24949b4322353b00e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fccb863bdf84b24949b4322353b00e7" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 33471 } } peers { permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 } } peers { permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 } } }
I20260923 20:15:20.015676 7091 tablet_replica.cc:442] T 470ae0e2eb5c448f8ede48b83600d3dd P 120ee2de42f24cf0b5cd4d39d141412e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:20.034145 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:20.027302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:33471 (local address 127.5.83.65:59459)
0923 20:15:20.027995 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:20.028014 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:20.028145 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:20.028523 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:20.028569 (+ 46us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:20.028587 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:15:20.028941 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:20.028950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.030447 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.030454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:20.031803 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:20.031814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.032881 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.032885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:20.033075 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:20.033134 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:15:20.033581 (+ 447us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:15:20.033596 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 20:15:20.033654 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 20:15:20.033766 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":2,"thread_start_us":399,"threads_started":1}
I20260923 20:15:20.034210 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:20.027138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:59459 (local address 127.5.83.65:33471)
0923 20:15:20.027318 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:20.027323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:20.028113 (+ 790us) server_negotiation.cc:407] Connection header received
0923 20:15:20.028276 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:20.028282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:20.028370 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:20.028518 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:15:20.029099 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.030313 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.031948 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.032706 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.032838 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:20.033306 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:15:20.033479 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:15:20.033577 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:20.033717 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 20:15:20.033809 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":74}
I20260923 20:15:20.044711 7019 raft_consensus.cc:1261] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e [term 1 FOLLOWER]: Refusing update from remote peer 6fccb863bdf84b24949b4322353b00e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:20.044744 6879 raft_consensus.cc:1261] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [term 1 FOLLOWER]: Refusing update from remote peer 6fccb863bdf84b24949b4322353b00e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:20.045285 7094 consensus_queue.cc:1035] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad8e4ce4f3a497cbff2228e1630992f" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:20.045583 7090 consensus_queue.cc:1035] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "120ee2de42f24cf0b5cd4d39d141412e" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:20.118312 6579 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:20.118556 6579 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:20.121727 6579 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6597b5c21f674c00bf9df83771af8c1d: Sending DeleteTablet for 3 replicas of tablet 604e0a297fc8449eb691e87be27e2ac3
I20260923 20:15:20.122514 6715 tablet_service.cc:1514] Processing DeleteTablet for tablet 604e0a297fc8449eb691e87be27e2ac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:20 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:41768
I20260923 20:15:20.122753 6855 tablet_service.cc:1514] Processing DeleteTablet for tablet 604e0a297fc8449eb691e87be27e2ac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:20 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:41290
I20260923 20:15:20.122851 6994 tablet_service.cc:1514] Processing DeleteTablet for tablet 604e0a297fc8449eb691e87be27e2ac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:20 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:42950
I20260923 20:15:20.123000 7108 tablet_replica.cc:330] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f: stopping tablet replica
I20260923 20:15:20.123172 7107 tablet_replica.cc:330] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7: stopping tablet replica
I20260923 20:15:20.123200 7108 raft_consensus.cc:2229] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:20.123385 7108 raft_consensus.cc:2258] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:20.123399 7107 raft_consensus.cc:2229] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:15:20.123720 7107 raft_consensus.cc:2258] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:20.124008 7106 tablet_replica.cc:330] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e: stopping tablet replica
I20260923 20:15:20.124290 7106 raft_consensus.cc:2229] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:20.124524 7106 raft_consensus.cc:2258] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:20.124655 7108 ts_tablet_manager.cc:1905] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:20.126446 7107 ts_tablet_manager.cc:1905] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:20.131860 7108 ts_tablet_manager.cc:1918] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:15:20.132004 7108 log.cc:1196] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/604e0a297fc8449eb691e87be27e2ac3
I20260923 20:15:20.132421 7108 ts_tablet_manager.cc:1939] T 604e0a297fc8449eb691e87be27e2ac3 P 9ad8e4ce4f3a497cbff2228e1630992f: Deleting consensus metadata
I20260923 20:15:20.133566 6566 catalog_manager.cc:4841] TS 9ad8e4ce4f3a497cbff2228e1630992f (127.5.83.66:43679): tablet 604e0a297fc8449eb691e87be27e2ac3 (table test-table [id=e241918788574003b490da9fa306358d]) successfully deleted
I20260923 20:15:20.134622 7106 ts_tablet_manager.cc:1905] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:20.137285 7107 ts_tablet_manager.cc:1918] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:15:20.137609 7107 log.cc:1196] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/604e0a297fc8449eb691e87be27e2ac3
I20260923 20:15:20.138221 7107 ts_tablet_manager.cc:1939] T 604e0a297fc8449eb691e87be27e2ac3 P 6fccb863bdf84b24949b4322353b00e7: Deleting consensus metadata
I20260923 20:15:20.139716 6564 catalog_manager.cc:4841] TS 6fccb863bdf84b24949b4322353b00e7 (127.5.83.65:33471): tablet 604e0a297fc8449eb691e87be27e2ac3 (table test-table [id=e241918788574003b490da9fa306358d]) successfully deleted
I20260923 20:15:20.141475 7106 ts_tablet_manager.cc:1918] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:15:20.141690 7106 log.cc:1196] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/604e0a297fc8449eb691e87be27e2ac3
I20260923 20:15:20.142162 7106 ts_tablet_manager.cc:1939] T 604e0a297fc8449eb691e87be27e2ac3 P 120ee2de42f24cf0b5cd4d39d141412e: Deleting consensus metadata
I20260923 20:15:20.143173 6567 catalog_manager.cc:4841] TS 120ee2de42f24cf0b5cd4d39d141412e (127.5.83.67:45167): tablet 604e0a297fc8449eb691e87be27e2ac3 (table test-table [id=e241918788574003b490da9fa306358d]) successfully deleted
Sep 23 20:15:20 dist-test-slave-vl24 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:20.160142 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:20.142739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.5.83.65:33471)
0923 20:15:20.142921 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:20.142929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:20.143830 (+ 901us) server_negotiation.cc:407] Connection header received
0923 20:15:20.144002 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:20.144007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:20.144085 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:20.144193 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:20.145698 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.146851 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.148172 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.148873 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.148999 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:20.153345 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:20.153361 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:20.153381 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:20.153427 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:20.157191 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:20.157879 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:20.157889 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:20.157892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:20.158042 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:20.158378 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:20.158383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:20.158387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:20.158896 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:20.159207 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:20.159381 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 20:15:20.159676 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260923 20:15:20.161101 6781 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41792
I20260923 20:15:20.177575 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:20.164107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.5.83.126:38837)
0923 20:15:20.164296 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:20.164303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:20.164572 (+ 269us) server_negotiation.cc:407] Connection header received
0923 20:15:20.164772 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:20.164779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:20.164906 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:20.165008 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:20.166187 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.167299 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.168481 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.169163 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.169275 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:20.171571 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:20.171578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:20.171582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:20.171632 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:20.174995 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:20.175647 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:20.175652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:20.175656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:20.175800 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:20.176118 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:20.176122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:20.176126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:20.176469 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:20.176770 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:20.177058 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 20:15:20.177263 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260923 20:15:20.178194 6579 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47122
I20260923 20:15:20.180380 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 6651
I20260923 20:15:20.196144 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:20.181795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.5.83.65:33471)
0923 20:15:20.181971 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:20.181977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:20.182206 (+ 229us) server_negotiation.cc:407] Connection header received
0923 20:15:20.182343 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:20.182349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:20.182425 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:20.182525 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:20.183680 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.184894 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.186159 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.186921 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.187041 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:20.190060 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:20.190065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:20.190068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:20.190100 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:20.193340 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:20.194114 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:20.194118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:20.194120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:20.194268 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:20.194649 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:20.194652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:20.194654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:20.194915 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:20.195099 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:20.195433 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 20:15:20.195599 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 20:15:20.196830 6781 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41796
W20260923 20:15:20.197395 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 6651: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:15:20.197613 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 6651
I20260923 20:15:20.219708 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 6790
Sep 23 20:15:20 dist-test-slave-vl24 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:15:20.234325 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:20.220382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.5.83.66:43679)
0923 20:15:20.220917 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:20.220921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:20.220938 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:15:20.220994 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:20.220999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:20.221063 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:20.221164 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:20.222305 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.223229 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.224588 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.225028 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.225133 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:20.229060 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:20.229072 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:20.229105 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:20.229136 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:20.231780 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:20.232350 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:20.232357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:20.232359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:20.232479 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:20.233034 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:20.233037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:20.233039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:20.233370 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:20.233586 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:20.233839 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 20:15:20.234013 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":308,"threads_started":1}
W20260923 20:15:20.235140 6920 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41302
W20260923 20:15:20.235776 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 6790: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:15:20.235908 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 6790
I20260923 20:15:20.253328 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 6929
Sep 23 20:15:20 dist-test-slave-vl24 krb5kdc[6535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194519, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:15:20.268151 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:20.254220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.5.83.67:45167)
0923 20:15:20.254394 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:20.254398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:20.254413 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:15:20.254467 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:20.254470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:20.254526 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:20.254618 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:20.255851 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.256654 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.257850 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:20.258279 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:20.258374 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:20.262582 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:20.262597 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:20.262614 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:20.262649 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:20.265765 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:20.266416 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:20.266423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:20.266426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:20.266548 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:20.266924 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:20.266927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:20.266929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:20.267287 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:20.267527 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:20.267765 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 20:15:20.267928 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 20:15:20.268904 7059 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42966
W20260923 20:15:20.269471 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 6929: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:15:20.269654 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 6929
I20260923 20:15:20.286638 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 6548
I20260923 20:15:20.287767 6620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:20.405030 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 6548
2026-09-23T20:15:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4376 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:15:20 dist-test-slave-vl24 krb5kdc[7145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:15:20 dist-test-slave-vl24 krb5kdc[7145](info): set up 2 sockets
Sep 23 20:15:20 dist-test-slave-vl24 krb5kdc[7145](info): commencing operation
krb5kdc: starting...
W20260923 20:15:21.493917 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:15:21 dist-test-slave-vl24 krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:15:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:15:21.683959 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:40277
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:40277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:22.005679 7158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:22.005965 7158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:22.006100 7158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:22.013447 7158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:22.013594 7158 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 20:15:22.013667 7158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:22.013774 7158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:22.013861 7158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:22.013973 7158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:15:22.023967 7158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:40277
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:40277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:22.025316 7158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:22.026646 7158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:22.038667 7163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:22.038614 7166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:22.038668 7164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:22.039530 7158 server_base.cc:1034] running on GCE node
I20260923 20:15:22.040283 7158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:22.041795 7158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:22.043028 7158 hybrid_clock.cc:648] HybridClock initialized: now 1790194522042966 us; error 31 us; skew 500 ppm
I20260923 20:15:22.043399 7158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:22 dist-test-slave-vl24 krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194522, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:22.046559 7158 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:15:22.048048 7158 webserver.cc:460] Webserver started at http://127.5.83.126:39167/ using document root <none> and password file <none>
I20260923 20:15:22.048840 7158 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:22.048954 7158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:22.049288 7158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:22.052366 7158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "3900417b35834f8ebeac761a61d9320a"
format_stamp: "Formatted at 2026-09-23 20:15:22 on dist-test-slave-vl24"
server_key: "c8e4e3aa5450b1d255b4282502d83105"
server_key_iv: "9c7c60e89f5c4058230e6160839e303e"
server_key_version: "encryptionkey@0"
I20260923 20:15:22.053193 7158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "3900417b35834f8ebeac761a61d9320a"
format_stamp: "Formatted at 2026-09-23 20:15:22 on dist-test-slave-vl24"
server_key: "c8e4e3aa5450b1d255b4282502d83105"
server_key_iv: "9c7c60e89f5c4058230e6160839e303e"
server_key_version: "encryptionkey@0"
I20260923 20:15:22.058329 7158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:15:22.061692 7173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:22.063051 7158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:22.063232 7158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "3900417b35834f8ebeac761a61d9320a"
format_stamp: "Formatted at 2026-09-23 20:15:22 on dist-test-slave-vl24"
server_key: "c8e4e3aa5450b1d255b4282502d83105"
server_key_iv: "9c7c60e89f5c4058230e6160839e303e"
server_key_version: "encryptionkey@0"
I20260923 20:15:22.063472 7158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:22.076458 7158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:22.080339 7158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:22.080638 7158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:22.102538 7158 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:40277
I20260923 20:15:22.102545 7234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:40277 every 8 connection(s)
I20260923 20:15:22.103909 7158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:22.107635 7235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:22.112799 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 7158
I20260923 20:15:22.113036 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:22.113636 5453 external_mini_cluster.cc:1428] Setting key e2cec9807e7a9bf87f9e020f28f21b2f
I20260923 20:15:22.118343 7235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a: Bootstrap starting.
I20260923 20:15:22.122148 7235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:22.123426 7235 log.cc:824] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:22.126601 7235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a: No bootstrap required, opened a new log
Sep 23 20:15:22 dist-test-slave-vl24 krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:22.135089 7235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3900417b35834f8ebeac761a61d9320a" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40277 } }
I20260923 20:15:22.135515 7235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:22.135653 7235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3900417b35834f8ebeac761a61d9320a, State: Initialized, Role: FOLLOWER
I20260923 20:15:22.136406 7235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3900417b35834f8ebeac761a61d9320a" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40277 } }
I20260923 20:15:22.136732 7235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:22.136888 7235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:22.137074 7235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:22.137987 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:22.115746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.5.83.126:40277)
0923 20:15:22.116681 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:22.116693 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:22.116747 (+ 54us) server_negotiation.cc:407] Connection header received
0923 20:15:22.118390 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:22.118414 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:22.118854 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:22.119186 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:22.121108 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:22.122439 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:22.124259 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:22.125096 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:22.125223 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:22.130044 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:22.130052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:22.130072 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:22.130114 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:22.133314 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:22.134122 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:22.134138 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:22.134146 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:22.134325 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:22.134706 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:22.134712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:22.134716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:22.135258 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:22.135581 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:22.135998 (+ 417us) server_negotiation.cc:299] Negotiation successful
0923 20:15:22.136477 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":431,"threads_started":1}
I20260923 20:15:22.139745 7235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3900417b35834f8ebeac761a61d9320a" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40277 } }
I20260923 20:15:22.140287 7235 leader_election.cc:302] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3900417b35834f8ebeac761a61d9320a; no voters:
I20260923 20:15:22.140967 7235 leader_election.cc:288] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:22.141096 7240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:22.141832 7240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [term 1 LEADER]: Becoming Leader. State: Replica: 3900417b35834f8ebeac761a61d9320a, State: Running, Role: LEADER
I20260923 20:15:22.142364 7240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3900417b35834f8ebeac761a61d9320a" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40277 } }
I20260923 20:15:22.143069 7235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:22.145210 7241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3900417b35834f8ebeac761a61d9320a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3900417b35834f8ebeac761a61d9320a" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40277 } } }
I20260923 20:15:22.145689 7241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:22.145372 7242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3900417b35834f8ebeac761a61d9320a. Latest consensus state: current_term: 1 leader_uuid: "3900417b35834f8ebeac761a61d9320a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3900417b35834f8ebeac761a61d9320a" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40277 } } }
I20260923 20:15:22.146101 7242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:22.146629 7247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:22.152218 7247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:22.162060 7247 catalog_manager.cc:1347] Generated new cluster ID: fad8f1d0d3e24cea979793318ba1a369
I20260923 20:15:22.162204 7247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:22.182358 7247 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:22.183907 7247 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:22.190670 7247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a: Generated new TSK 0
I20260923 20:15:22.191481 7247 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:15:22.239854 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:40277
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:22.566430 7261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:22.566799 7261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:22.566970 7261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:22.574466 7261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:22.574582 7261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:22.574723 7261 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 20:15:22.574800 7261 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:15:22.574934 7261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:15:22.585112 7261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.126:40277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:22.586457 7261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:22.587817 7261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:22.601279 7266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:22.601276 7269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:22.601312 7267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:22.602095 7261 server_base.cc:1034] running on GCE node
I20260923 20:15:22.602766 7261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:22.603746 7261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:22.604990 7261 hybrid_clock.cc:648] HybridClock initialized: now 1790194522604944 us; error 83 us; skew 500 ppm
I20260923 20:15:22.605322 7261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:22 dist-test-slave-vl24 krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194522, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:22.608407 7261 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:15:22.609992 7261 webserver.cc:460] Webserver started at http://127.5.83.65:44695/ using document root <none> and password file <none>
I20260923 20:15:22.610683 7261 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:22.610826 7261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:22.611146 7261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:22.614157 7261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "adf4cfe3807442ec90541fca3b8d471d"
format_stamp: "Formatted at 2026-09-23 20:15:22 on dist-test-slave-vl24"
server_key: "2d1737ecd328459737b9e2c6db295d8d"
server_key_iv: "2fcc08c2eeafebc7a3eb95be10902d1a"
server_key_version: "encryptionkey@0"
I20260923 20:15:22.614981 7261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "adf4cfe3807442ec90541fca3b8d471d"
format_stamp: "Formatted at 2026-09-23 20:15:22 on dist-test-slave-vl24"
server_key: "2d1737ecd328459737b9e2c6db295d8d"
server_key_iv: "2fcc08c2eeafebc7a3eb95be10902d1a"
server_key_version: "encryptionkey@0"
I20260923 20:15:22.620311 7261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:15:22.623879 7276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:22.625308 7261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:22.625489 7261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "adf4cfe3807442ec90541fca3b8d471d"
format_stamp: "Formatted at 2026-09-23 20:15:22 on dist-test-slave-vl24"
server_key: "2d1737ecd328459737b9e2c6db295d8d"
server_key_iv: "2fcc08c2eeafebc7a3eb95be10902d1a"
server_key_version: "encryptionkey@0"
I20260923 20:15:22.625694 7261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:22.648970 7261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:22.653566 7261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:22.653872 7261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:22.656643 7261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:22.656798 7261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:22.656942 7261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:22.657083 7261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:15:22 dist-test-slave-vl24 krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194522, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:22.681648 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:22.657169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:40465 (local address 127.5.83.126:40277)
0923 20:15:22.657621 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:22.657627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:22.659007 (+ 1380us) server_negotiation.cc:407] Connection header received
0923 20:15:22.661845 (+ 2838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:22.661853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:22.661955 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:22.662092 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:22.664860 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:22.665595 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:22.668185 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:22.668671 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:22.668805 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:22.673931 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:22.673940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:22.673946 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:22.673992 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:22.677455 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:22.678606 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:22.678614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:22.678620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:22.678776 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:22.679357 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:22.679366 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:22.679372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:22.679862 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:22.680045 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:22.680785 (+ 740us) server_negotiation.cc:299] Negotiation successful
0923 20:15:22.681093 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":224,"threads_started":1}
I20260923 20:15:22.682200 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:22.657863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:40277 (local address 127.5.83.65:40465)
0923 20:15:22.658700 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:22.658763 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:22.661469 (+ 2706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:22.662396 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:22.662406 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:22.663393 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:22.664605 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:22.664643 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:22.665757 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:22.665769 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:22.668023 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:22.668042 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:22.668915 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:22.668923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:22.669237 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:22.670106 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:22.670157 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:22.673648 (+ 3491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:22.677605 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:22.677621 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:22.677643 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:22.678441 (+ 798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:22.678928 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:22.678934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:22.678937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:22.679168 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:22.680012 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:22.680018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:22.680345 (+ 327us) client_negotiation.cc:769] Sending connection context
0923 20:15:22.680652 (+ 307us) client_negotiation.cc:240] Negotiation successful
0923 20:15:22.681034 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":299,"threads_started":1}
I20260923 20:15:22.704243 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:22.687731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:40277 (local address 127.5.83.65:43781)
0923 20:15:22.689582 (+ 1851us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:22.689607 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:22.689789 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:22.690172 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:22.690178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:22.690936 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:22.691388 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:22.691403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:22.692371 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:22.692379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:22.694564 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:22.694583 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:22.695538 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:22.695542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:22.695694 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:22.696471 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:22.696488 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:22.697781 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:22.701448 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:22.701454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:22.701459 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:22.701911 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:22.702280 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:22.702285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:22.702289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:22.702421 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:22.703086 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:22.703092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:22.703350 (+ 258us) client_negotiation.cc:769] Sending connection context
0923 20:15:22.703528 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 20:15:22.703907 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1654,"thread_start_us":1406,"threads_started":1}
I20260923 20:15:22.704793 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:22.687830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:43781 (local address 127.5.83.126:40277)
0923 20:15:22.688070 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:22.688078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:22.689710 (+ 1632us) server_negotiation.cc:407] Connection header received
0923 20:15:22.689922 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:22.689925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:22.689986 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:22.690098 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:22.691542 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:22.692225 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:22.694722 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:22.695100 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:22.695257 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:22.697973 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:22.697978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:22.697982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:22.698038 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:22.701273 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:22.702039 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:22.702043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:22.702046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:22.702169 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:22.702546 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:22.702558 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:22.702560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:22.702953 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:22.703113 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:22.703999 (+ 886us) server_negotiation.cc:299] Negotiation successful
0923 20:15:22.704314 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:15:22.711581 7261 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:34103
I20260923 20:15:22.711663 7396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:34103 every 8 connection(s)
I20260923 20:15:22.712940 7261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:15:22.714920 7397 heartbeater.cc:347] Connected to a master server at 127.5.83.126:40277
I20260923 20:15:22.715150 7397 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:22.715772 7397 heartbeater.cc:511] Master 127.5.83.126:40277 requested a full tablet report, sending...
I20260923 20:15:22.717672 7190 ts_manager.cc:194] Registered new tserver with Master: adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103)
I20260923 20:15:22.719095 7190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:40465
I20260923 20:15:22.720384 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 7261
I20260923 20:15:22.720629 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:15:22.721222 5453 external_mini_cluster.cc:1428] Setting key 073d1dc6f9026fbd1d93c8ecf10377a7
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:15:22.759868 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:40277
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:23.079761 7401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:23.080034 7401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:23.080149 7401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:23.087550 7401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:23.087662 7401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:23.087751 7401 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 20:15:23.087803 7401 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:15:23.087865 7401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:15:23.097683 7401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.126:40277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:23.098963 7401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:23.100245 7401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:23.114886 7406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:23.114914 7407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:23.115041 7409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:23.115536 7401 server_base.cc:1034] running on GCE node
I20260923 20:15:23.116240 7401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:23.117306 7401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:23.118528 7401 hybrid_clock.cc:648] HybridClock initialized: now 1790194523118475 us; error 49 us; skew 500 ppm
I20260923 20:15:23.118800 7401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:23 dist-test-slave-vl24 krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194523, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:23.121873 7401 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:15:23.123317 7401 webserver.cc:460] Webserver started at http://127.5.83.66:35385/ using document root <none> and password file <none>
I20260923 20:15:23.123951 7401 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:23.124059 7401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:23.124310 7401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:23.127271 7401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "ea87e29a4b134c0c9141da03dec72d00"
format_stamp: "Formatted at 2026-09-23 20:15:23 on dist-test-slave-vl24"
server_key: "67ac279fbdd7240610d2695d7cb24212"
server_key_iv: "3f6479cb0fd08e10b151e21013cce052"
server_key_version: "encryptionkey@0"
I20260923 20:15:23.127998 7401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "ea87e29a4b134c0c9141da03dec72d00"
format_stamp: "Formatted at 2026-09-23 20:15:23 on dist-test-slave-vl24"
server_key: "67ac279fbdd7240610d2695d7cb24212"
server_key_iv: "3f6479cb0fd08e10b151e21013cce052"
server_key_version: "encryptionkey@0"
I20260923 20:15:23.133327 7401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 20:15:23.136899 7416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:23.138253 7401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:15:23.138384 7401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "ea87e29a4b134c0c9141da03dec72d00"
format_stamp: "Formatted at 2026-09-23 20:15:23 on dist-test-slave-vl24"
server_key: "67ac279fbdd7240610d2695d7cb24212"
server_key_iv: "3f6479cb0fd08e10b151e21013cce052"
server_key_version: "encryptionkey@0"
I20260923 20:15:23.138549 7401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:23.154306 7401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:23.158435 7401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:23.158728 7401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:23.161667 7401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:23.161841 7401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 20:15:23.161995 7401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:23.162132 7401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:15:23 dist-test-slave-vl24 krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194523, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:23.186314 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.165499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:53755 (local address 127.5.83.126:40277)
0923 20:15:23.165724 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:23.165730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:23.165755 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:15:23.167439 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:23.167447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:23.167589 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:23.167702 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:23.170063 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.171248 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.174249 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.174890 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.174969 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.179867 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:23.179877 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:23.179885 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:23.179935 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:23.183201 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:23.183929 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:23.183934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:23.183938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:23.184062 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:23.184509 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:23.184513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:23.184516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:23.184900 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:23.185049 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:23.185751 (+ 702us) server_negotiation.cc:299] Negotiation successful
0923 20:15:23.185968 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 20:15:23.187400 7425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.163098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:40277 (local address 127.5.83.66:53755)
0923 20:15:23.164394 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:23.164478 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:23.167093 (+ 2615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:23.168097 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:23.168109 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:23.168994 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:23.169896 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.169914 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.171391 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.171397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.174032 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.174054 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.175275 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.175283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.175656 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.176476 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:23.176501 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:23.179607 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:23.183361 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:23.183370 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:23.183386 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:23.183799 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:23.184191 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:23.184197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:23.184200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:23.184386 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:23.185020 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:23.185027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:23.185378 (+ 351us) client_negotiation.cc:769] Sending connection context
0923 20:15:23.185634 (+ 256us) client_negotiation.cc:240] Negotiation successful
0923 20:15:23.186146 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":536,"threads_started":1}
I20260923 20:15:23.207772 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.193288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:40731 (local address 127.5.83.126:40277)
0923 20:15:23.193453 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:23.193457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:23.194109 (+ 652us) server_negotiation.cc:407] Connection header received
0923 20:15:23.194305 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:23.194310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:23.194376 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:23.194473 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:23.195349 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.196245 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.198111 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.198747 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.198922 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.201578 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:23.201583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:23.201586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:23.201616 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:23.204118 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:23.205148 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:23.205152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:23.205154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:23.205294 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:23.205769 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:23.205774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:23.205776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:23.206231 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:23.206399 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:23.207046 (+ 647us) server_negotiation.cc:299] Negotiation successful
0923 20:15:23.207333 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:15:23.208209 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.193366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:40277 (local address 127.5.83.66:40731)
0923 20:15:23.194002 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:23.194023 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:23.194176 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:23.194497 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:23.194501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:23.194933 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:23.195210 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.195219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.196390 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.196395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.197899 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.197918 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.198996 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.199004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.199327 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.200032 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:23.200062 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:23.201384 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:23.204258 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:23.204267 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:23.204274 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:23.205017 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:23.205404 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:23.205414 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:23.205418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:23.205623 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:23.206366 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:23.206371 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:23.206643 (+ 272us) client_negotiation.cc:769] Sending connection context
0923 20:15:23.206892 (+ 249us) client_negotiation.cc:240] Negotiation successful
0923 20:15:23.207406 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":375,"threads_started":1}
I20260923 20:15:23.225662 7401 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:43953
I20260923 20:15:23.225710 7535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:43953 every 8 connection(s)
I20260923 20:15:23.226955 7401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:15:23.229110 7536 heartbeater.cc:347] Connected to a master server at 127.5.83.126:40277
I20260923 20:15:23.229379 7536 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:23.229987 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 7401
I20260923 20:15:23.230165 7536 heartbeater.cc:511] Master 127.5.83.126:40277 requested a full tablet report, sending...
I20260923 20:15:23.230207 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:15:23.230631 5453 external_mini_cluster.cc:1428] Setting key 4d860db597fd0e2c3af8437756986838
I20260923 20:15:23.231745 7190 ts_manager.cc:194] Registered new tserver with Master: ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953)
I20260923 20:15:23.232806 7190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:53755
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:15:23.272153 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:40277
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:23.613878 7540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:23.614274 7540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:23.614464 7540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:23.622152 7540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:23.622310 7540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:23.622471 7540 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 20:15:23.622565 7540 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:15:23.622706 7540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:15:23.632979 7540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.126:40277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:23.634344 7540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:23.635764 7540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:23.649616 7545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:23.649530 7546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:23.649591 7548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:23.653977 7540 server_base.cc:1034] running on GCE node
I20260923 20:15:23.654640 7540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:23.655604 7540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:23.656838 7540 hybrid_clock.cc:648] HybridClock initialized: now 1790194523656783 us; error 34 us; skew 500 ppm
I20260923 20:15:23.657243 7540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:23 dist-test-slave-vl24 krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194523, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:23.660511 7540 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:15:23.662142 7540 webserver.cc:460] Webserver started at http://127.5.83.67:42437/ using document root <none> and password file <none>
I20260923 20:15:23.662889 7540 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:23.663053 7540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:23.663384 7540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:23.666529 7540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "1eb80d1e42034d02b0a7873c2e2b8405"
format_stamp: "Formatted at 2026-09-23 20:15:23 on dist-test-slave-vl24"
server_key: "b25a2cf312f5dfe915b804aa76bac23e"
server_key_iv: "3678f02db12f99408c8c32af09e38c17"
server_key_version: "encryptionkey@0"
I20260923 20:15:23.667505 7540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "1eb80d1e42034d02b0a7873c2e2b8405"
format_stamp: "Formatted at 2026-09-23 20:15:23 on dist-test-slave-vl24"
server_key: "b25a2cf312f5dfe915b804aa76bac23e"
server_key_iv: "3678f02db12f99408c8c32af09e38c17"
server_key_version: "encryptionkey@0"
I20260923 20:15:23.673120 7540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:15:23.676689 7555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:23.678037 7540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:15:23.678231 7540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "1eb80d1e42034d02b0a7873c2e2b8405"
format_stamp: "Formatted at 2026-09-23 20:15:23 on dist-test-slave-vl24"
server_key: "b25a2cf312f5dfe915b804aa76bac23e"
server_key_iv: "3678f02db12f99408c8c32af09e38c17"
server_key_version: "encryptionkey@0"
I20260923 20:15:23.678448 7540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:23.699695 7540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:23.703944 7540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:23.704248 7540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:23.707008 7540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:23.707157 7540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:23.707297 7540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:23.707389 7540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:23.722074 7397 heartbeater.cc:503] Master 127.5.83.126:40277 was elected leader, sending a full tablet report...
Sep 23 20:15:23 dist-test-slave-vl24 krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194523, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:23.733264 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.707938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:34245 (local address 127.5.83.126:40277)
0923 20:15:23.708161 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:23.708167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:23.709944 (+ 1777us) server_negotiation.cc:407] Connection header received
0923 20:15:23.712947 (+ 3003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:23.712954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:23.713044 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:23.713163 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:23.715944 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.717092 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.719925 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.720625 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.720809 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.724909 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:23.724931 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:23.724938 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:23.724987 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:23.728825 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:23.729878 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:23.729886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:23.729892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:23.730065 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:23.730661 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:23.730669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:23.730672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:23.731079 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:23.731240 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:23.732400 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0923 20:15:23.732753 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:15:23.734134 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.708801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:40277 (local address 127.5.83.67:34245)
0923 20:15:23.709751 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:23.709805 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:23.712587 (+ 2782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:23.713586 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:23.713600 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:23.714830 (+ 1230us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:23.715698 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.715725 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.717386 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.717399 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.719750 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.719775 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.720929 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.720940 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.721182 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.721993 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:23.722021 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:23.724683 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:23.728983 (+ 4300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:23.728995 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:23.729020 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:23.729762 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:23.730234 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:23.730241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:23.730247 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:23.730532 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:23.731316 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:23.731321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:23.731863 (+ 542us) client_negotiation.cc:769] Sending connection context
0923 20:15:23.732233 (+ 370us) client_negotiation.cc:240] Negotiation successful
0923 20:15:23.732870 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":321,"threads_started":1}
I20260923 20:15:23.755949 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.740072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:51921 (local address 127.5.83.126:40277)
0923 20:15:23.740308 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:23.740317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:23.740958 (+ 641us) server_negotiation.cc:407] Connection header received
0923 20:15:23.741173 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:23.741179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:23.741266 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:23.741368 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:23.742353 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.743320 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.744827 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.745522 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.745684 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.748408 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:23.748413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:23.748416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:23.748456 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:23.752479 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:23.753702 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:23.753710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:23.753716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:23.753895 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:23.754421 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:23.754429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:23.754433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:23.754727 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:23.754887 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:23.755306 (+ 419us) server_negotiation.cc:299] Negotiation successful
0923 20:15:23.755563 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 20:15:23.756021 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.740104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:40277 (local address 127.5.83.67:51921)
0923 20:15:23.740834 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:23.740861 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:23.741042 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:23.741393 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:23.741397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:23.741870 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:23.742217 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.742227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.743457 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.743465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.744694 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.744705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.745774 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.745788 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.746077 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.746826 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:23.746858 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:23.748182 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:23.752700 (+ 4518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:23.752711 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:23.752718 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:23.753479 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:23.754073 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:23.754079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:23.754088 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:23.754262 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:23.754862 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:23.754867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:23.755058 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 20:15:23.755212 (+ 154us) client_negotiation.cc:240] Negotiation successful
0923 20:15:23.755556 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":434,"threads_started":1}
I20260923 20:15:23.767378 7540 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:38031
I20260923 20:15:23.767438 7674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:38031 every 8 connection(s)
I20260923 20:15:23.768738 7540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:15:23.771004 7675 heartbeater.cc:347] Connected to a master server at 127.5.83.126:40277
I20260923 20:15:23.771045 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 7540
I20260923 20:15:23.771241 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:15:23.771252 7675 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:23.771730 5453 external_mini_cluster.cc:1428] Setting key 987006d938dff5c33f922e805c90e814
I20260923 20:15:23.771955 7675 heartbeater.cc:511] Master 127.5.83.126:40277 requested a full tablet report, sending...
I20260923 20:15:23.773571 7190 ts_manager.cc:194] Registered new tserver with Master: 1eb80d1e42034d02b0a7873c2e2b8405 (127.5.83.67:38031)
I20260923 20:15:23.774525 7190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:34245
I20260923 20:15:23.774971 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:15:23 dist-test-slave-vl24 krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:15:23 dist-test-slave-vl24 krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:23.821835 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.807858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.5.83.126:40277)
0923 20:15:23.808128 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:23.808136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:23.808453 (+ 317us) server_negotiation.cc:407] Connection header received
0923 20:15:23.808733 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:23.808738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:23.808803 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:23.808914 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:23.810292 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.811018 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.812265 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.812756 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.812901 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.816667 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:23.816673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:23.816676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:23.816729 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:23.819288 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:23.819878 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:23.819882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:23.819885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:23.819998 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:23.820329 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:23.820332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:23.820334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:23.820687 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:23.820860 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:23.821234 (+ 374us) server_negotiation.cc:299] Negotiation successful
0923 20:15:23.821429 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 20:15:23.825923 7190 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:15:23.833124 7190 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:15:23.872694 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.846281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48618 (local address 127.5.83.65:34103)
0923 20:15:23.861734 (+ 15453us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:23.861748 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:23.863851 (+ 2103us) server_negotiation.cc:407] Connection header received
0923 20:15:23.864136 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:23.864144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:23.864376 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:23.864599 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:23.865660 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.866921 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.869602 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.870880 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.871015 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.871229 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:23.871568 (+ 339us) server_negotiation.cc:299] Negotiation successful
0923 20:15:23.872316 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15182,"thread_start_us":16030,"threads_started":1}
I20260923 20:15:23.872872 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.846085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:34103 (local address 127.0.0.1:48618)
0923 20:15:23.863652 (+ 17567us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:23.863676 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:23.863877 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:23.864832 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:23.864840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:23.864889 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:23.865433 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.865445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.867034 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.867040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.869443 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.869454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.871083 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.871089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.871294 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.871345 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 20:15:23.871476 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 20:15:23.871663 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17393,"mutex_wait_us":86,"spinlock_wait_cycles":1408,"thread_start_us":17531,"threads_started":1}
I20260923 20:15:23.873382 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.846299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.5.83.67:38031)
0923 20:15:23.862365 (+ 16066us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:23.862377 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:23.863705 (+ 1328us) server_negotiation.cc:407] Connection header received
0923 20:15:23.863818 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:23.863827 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:23.864139 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:23.864329 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:23.865593 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.867249 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.870329 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.871473 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.871583 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.871809 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:23.872968 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0923 20:15:23.873141 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15842,"thread_start_us":15678,"threads_started":1}
I20260923 20:15:23.873799 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.846031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:38031 (local address 127.0.0.1:40630)
0923 20:15:23.863189 (+ 17158us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:23.863235 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:23.863472 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:23.864741 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:23.864748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:23.864777 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:23.865324 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.865341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.867444 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.867452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.870180 (+ 2728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.870195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.872014 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.872020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.872303 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.872365 (+ 62us) client_negotiation.cc:769] Sending connection context
0923 20:15:23.872440 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 20:15:23.872631 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16850,"mutex_wait_us":69,"thread_start_us":17988,"threads_started":1}
I20260923 20:15:23.876798 7327 tablet_service.cc:1467] Processing CreateTablet for tablet ac83dc68e29b4d4db44c0fd72ef684de (DEFAULT_TABLE table=test-table [id=2c8f2c20f79648cd8f991dedaa07b748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:23.877081 7605 tablet_service.cc:1467] Processing CreateTablet for tablet ac83dc68e29b4d4db44c0fd72ef684de (DEFAULT_TABLE table=test-table [id=2c8f2c20f79648cd8f991dedaa07b748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:23.878150 7327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac83dc68e29b4d4db44c0fd72ef684de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:23.878305 7605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac83dc68e29b4d4db44c0fd72ef684de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:23.881086 7688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.848416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:43953 (local address 127.0.0.1:41180)
0923 20:15:23.864329 (+ 15913us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:23.864346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:23.864457 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:23.865394 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:23.865405 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:23.865433 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:23.865883 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.865895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.867693 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.867702 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.869966 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:23.869980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.871992 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.872003 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:23.872309 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.872374 (+ 65us) client_negotiation.cc:769] Sending connection context
0923 20:15:23.872460 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 20:15:23.880712 (+ 8252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15780,"mutex_wait_us":76,"thread_start_us":14490,"threads_started":1}
I20260923 20:15:23.881670 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:23.849106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.5.83.66:43953)
0923 20:15:23.863053 (+ 13947us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:23.863065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:23.864618 (+ 1553us) server_negotiation.cc:407] Connection header received
0923 20:15:23.864723 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:23.864730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:23.864960 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:23.865156 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:23.866088 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.867506 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.870156 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:23.871432 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:23.871586 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:23.871777 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:23.872659 (+ 882us) server_negotiation.cc:299] Negotiation successful
0923 20:15:23.872847 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13729,"thread_start_us":13569,"threads_started":1}
I20260923 20:15:23.885352 7471 tablet_service.cc:1467] Processing CreateTablet for tablet ac83dc68e29b4d4db44c0fd72ef684de (DEFAULT_TABLE table=test-table [id=2c8f2c20f79648cd8f991dedaa07b748]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:23.886471 7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac83dc68e29b4d4db44c0fd72ef684de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:23.891278 7694 tablet_bootstrap.cc:492] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405: Bootstrap starting.
I20260923 20:15:23.894465 7695 tablet_bootstrap.cc:492] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d: Bootstrap starting.
I20260923 20:15:23.895810 7694 tablet_bootstrap.cc:654] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:23.897157 7694 log.cc:824] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:23.898087 7695 tablet_bootstrap.cc:654] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:23.899295 7695 log.cc:824] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:23.899634 7697 tablet_bootstrap.cc:492] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00: Bootstrap starting.
I20260923 20:15:23.900501 7694 tablet_bootstrap.cc:492] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405: No bootstrap required, opened a new log
I20260923 20:15:23.900895 7694 ts_tablet_manager.cc:1397] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260923 20:15:23.902761 7697 tablet_bootstrap.cc:654] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:23.903218 7695 tablet_bootstrap.cc:492] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d: No bootstrap required, opened a new log
I20260923 20:15:23.903565 7695 ts_tablet_manager.cc:1397] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260923 20:15:23.904508 7697 log.cc:824] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:23.907195 7697 tablet_bootstrap.cc:492] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00: No bootstrap required, opened a new log
I20260923 20:15:23.907460 7697 ts_tablet_manager.cc:1397] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260923 20:15:23.911665 7694 raft_consensus.cc:348] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } }
I20260923 20:15:23.912132 7694 raft_consensus.cc:374] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:23.912277 7694 raft_consensus.cc:729] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb80d1e42034d02b0a7873c2e2b8405, State: Initialized, Role: FOLLOWER
I20260923 20:15:23.913025 7694 consensus_queue.cc:260] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } }
I20260923 20:15:23.914273 7695 raft_consensus.cc:348] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } }
I20260923 20:15:23.914788 7695 raft_consensus.cc:374] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:23.914948 7694 ts_tablet_manager.cc:1428] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260923 20:15:23.914949 7695 raft_consensus.cc:729] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf4cfe3807442ec90541fca3b8d471d, State: Initialized, Role: FOLLOWER
I20260923 20:15:23.915531 7675 heartbeater.cc:503] Master 127.5.83.126:40277 was elected leader, sending a full tablet report...
I20260923 20:15:23.915618 7697 raft_consensus.cc:348] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } }
I20260923 20:15:23.916010 7697 raft_consensus.cc:374] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:23.916136 7697 raft_consensus.cc:729] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea87e29a4b134c0c9141da03dec72d00, State: Initialized, Role: FOLLOWER
I20260923 20:15:23.916811 7697 consensus_queue.cc:260] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } }
I20260923 20:15:23.917083 7695 consensus_queue.cc:260] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } }
I20260923 20:15:23.918056 7536 heartbeater.cc:503] Master 127.5.83.126:40277 was elected leader, sending a full tablet report...
I20260923 20:15:23.918632 7697 ts_tablet_manager.cc:1428] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260923 20:15:23.922075 7695 ts_tablet_manager.cc:1428] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d: Time spent starting tablet: real 0.018s user 0.010s sys 0.005s
W20260923 20:15:23.965328 7398 tablet.cc:2369] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:15:23.978983 7537 tablet.cc:2369] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:15:24.020390 7676 tablet.cc:2369] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:15:24.221040 7700 raft_consensus.cc:482] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:24.221330 7700 raft_consensus.cc:504] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } }
I20260923 20:15:24.222971 7700 leader_election.cc:288] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103), ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953)
I20260923 20:15:24.230311 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.223450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:34103 (local address 127.5.83.67:53671)
0923 20:15:24.223688 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:24.223714 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:24.223851 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:24.224299 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:24.224303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:24.224325 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:24.224770 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.224779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.226052 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.226057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.227480 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.227499 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.229429 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.229435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.229653 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.229691 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 20:15:24.229820 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 20:15:24.229986 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"spinlock_wait_cycles":429952}
I20260923 20:15:24.231170 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.223476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:53671 (local address 127.5.83.65:34103)
0923 20:15:24.223736 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:24.223742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:24.223772 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:15:24.224041 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:24.224048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:24.224138 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:24.224262 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:24.224916 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.225919 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.227673 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.229072 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.229216 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.229329 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:24.230740 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0923 20:15:24.230883 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260923 20:15:24.231367 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.223556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:36811 (local address 127.5.83.66:43953)
0923 20:15:24.223736 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:24.223742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:24.224408 (+ 666us) server_negotiation.cc:407] Connection header received
0923 20:15:24.224845 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:24.224873 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:24.224958 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:24.225057 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:24.225622 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.226894 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.228737 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.229995 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.230141 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.230249 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:24.230972 (+ 723us) server_negotiation.cc:299] Negotiation successful
0923 20:15:24.231108 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:15:24.231422 7703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.223495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:43953 (local address 127.5.83.67:36811)
0923 20:15:24.224286 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:24.224304 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:24.224645 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:24.225090 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:24.225095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:24.225119 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:24.225449 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.225460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.227054 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.227059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.228496 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.228507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.230651 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.230659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.230816 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.230839 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:15:24.230947 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:15:24.231085 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":492,"threads_started":1}
I20260923 20:15:24.232146 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac83dc68e29b4d4db44c0fd72ef684de" candidate_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf4cfe3807442ec90541fca3b8d471d" is_pre_election: true
I20260923 20:15:24.232708 7352 raft_consensus.cc:2454] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb80d1e42034d02b0a7873c2e2b8405 in term 0.
I20260923 20:15:24.233525 7557 leader_election.cc:302] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb80d1e42034d02b0a7873c2e2b8405, adf4cfe3807442ec90541fca3b8d471d; no voters:
I20260923 20:15:24.233794 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac83dc68e29b4d4db44c0fd72ef684de" candidate_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea87e29a4b134c0c9141da03dec72d00" is_pre_election: true
I20260923 20:15:24.234004 7700 raft_consensus.cc:2781] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:15:24.234154 7700 raft_consensus.cc:482] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:15:24.234277 7700 raft_consensus.cc:3037] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:24.234391 7491 raft_consensus.cc:2454] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb80d1e42034d02b0a7873c2e2b8405 in term 0.
I20260923 20:15:24.237833 7700 raft_consensus.cc:504] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } }
I20260923 20:15:24.238641 7700 leader_election.cc:288] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [CANDIDATE]: Term 1 election: Requested vote from peers adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103), ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953)
I20260923 20:15:24.239156 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac83dc68e29b4d4db44c0fd72ef684de" candidate_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea87e29a4b134c0c9141da03dec72d00"
I20260923 20:15:24.239178 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac83dc68e29b4d4db44c0fd72ef684de" candidate_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf4cfe3807442ec90541fca3b8d471d"
I20260923 20:15:24.239395 7491 raft_consensus.cc:3037] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:24.239387 7352 raft_consensus.cc:3037] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:24.241804 7491 raft_consensus.cc:2454] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb80d1e42034d02b0a7873c2e2b8405 in term 1.
I20260923 20:15:24.242254 7558 leader_election.cc:302] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb80d1e42034d02b0a7873c2e2b8405, ea87e29a4b134c0c9141da03dec72d00; no voters:
I20260923 20:15:24.242631 7700 raft_consensus.cc:2781] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:24.242852 7352 raft_consensus.cc:2454] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb80d1e42034d02b0a7873c2e2b8405 in term 1.
I20260923 20:15:24.243386 7700 raft_consensus.cc:686] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 LEADER]: Becoming Leader. State: Replica: 1eb80d1e42034d02b0a7873c2e2b8405, State: Running, Role: LEADER
I20260923 20:15:24.243918 7700 consensus_queue.cc:237] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } }
I20260923 20:15:24.249025 7189 catalog_manager.cc:5495] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb80d1e42034d02b0a7873c2e2b8405 (127.5.83.67). New cstate: current_term: 1 leader_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } health_report { overall_health: UNKNOWN } } }
Sep 23 20:15:24 dist-test-slave-vl24 krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194522, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:24.366619 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.346878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:40277 (local address 127.0.0.1:35946)
0923 20:15:24.347712 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:24.347741 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:24.347867 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:24.348188 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:24.348191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:24.349038 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:24.349300 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.349308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.350431 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.350434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.351378 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.351388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.352110 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.352114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.352235 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.358547 (+ 6312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:24.358567 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:24.360886 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:24.363989 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:24.364000 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:24.364005 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:24.364466 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:24.364887 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:24.364898 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:24.364901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:24.365021 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:24.365541 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:24.365549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:24.365710 (+ 161us) client_negotiation.cc:769] Sending connection context
0923 20:15:24.365866 (+ 156us) client_negotiation.cc:240] Negotiation successful
0923 20:15:24.366160 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":516,"threads_started":1}
I20260923 20:15:24.366679 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.346851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.5.83.126:40277)
0923 20:15:24.347436 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:24.347441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:24.347820 (+ 379us) server_negotiation.cc:407] Connection header received
0923 20:15:24.347966 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:24.347969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:24.348028 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:24.348127 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:24.349458 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.350317 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.351505 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.351927 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.352024 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.361112 (+ 9088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:24.361117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:24.361120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:24.361152 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:24.363733 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:24.364641 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:24.364649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:24.364652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:24.364771 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:24.365139 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:24.365144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:24.365147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:24.365418 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:24.365563 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:24.365956 (+ 393us) server_negotiation.cc:299] Negotiation successful
0923 20:15:24.366305 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":471,"thread_start_us":333,"threads_started":1}
I20260923 20:15:24.371634 7189 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:35946:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 20:15:24.372782 7189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:15:24.379457 7471 tablet_service.cc:1467] Processing CreateTablet for tablet b8afec888c25441cabec8b5bb115f700 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac7f9cdc7333499799284071b32dfe3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:15:24.379457 7327 tablet_service.cc:1467] Processing CreateTablet for tablet b8afec888c25441cabec8b5bb115f700 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac7f9cdc7333499799284071b32dfe3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:15:24.379853 7605 tablet_service.cc:1467] Processing CreateTablet for tablet b8afec888c25441cabec8b5bb115f700 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac7f9cdc7333499799284071b32dfe3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:15:24.380656 7327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8afec888c25441cabec8b5bb115f700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:24.380658 7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8afec888c25441cabec8b5bb115f700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:24.380764 7605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8afec888c25441cabec8b5bb115f700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:24.386986 7694 tablet_bootstrap.cc:492] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405: Bootstrap starting.
I20260923 20:15:24.387970 7695 tablet_bootstrap.cc:492] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d: Bootstrap starting.
I20260923 20:15:24.388381 7697 tablet_bootstrap.cc:492] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: Bootstrap starting.
I20260923 20:15:24.388870 7694 tablet_bootstrap.cc:654] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:24.389771 7695 tablet_bootstrap.cc:654] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:24.390343 7697 tablet_bootstrap.cc:654] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:24.391146 7694 tablet_bootstrap.cc:492] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405: No bootstrap required, opened a new log
I20260923 20:15:24.391317 7694 ts_tablet_manager.cc:1397] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260923 20:15:24.392100 7695 tablet_bootstrap.cc:492] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d: No bootstrap required, opened a new log
I20260923 20:15:24.392252 7695 ts_tablet_manager.cc:1397] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 20:15:24.392134 7694 raft_consensus.cc:348] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.392385 7694 raft_consensus.cc:374] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:24.392486 7694 raft_consensus.cc:729] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb80d1e42034d02b0a7873c2e2b8405, State: Initialized, Role: FOLLOWER
I20260923 20:15:24.392759 7694 consensus_queue.cc:260] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.393354 7694 ts_tablet_manager.cc:1428] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260923 20:15:24.393204 7700 tablet_replica.cc:439] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } }
I20260923 20:15:24.393498 7700 tablet_replica.cc:442] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:24.393782 7695 raft_consensus.cc:348] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.394069 7695 raft_consensus.cc:374] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:24.394191 7695 raft_consensus.cc:729] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf4cfe3807442ec90541fca3b8d471d, State: Initialized, Role: FOLLOWER
I20260923 20:15:24.394493 7695 consensus_queue.cc:260] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.395035 7697 tablet_bootstrap.cc:492] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: No bootstrap required, opened a new log
I20260923 20:15:24.395201 7697 ts_tablet_manager.cc:1397] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260923 20:15:24.394958 7702 tablet_replica.cc:439] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } }
I20260923 20:15:24.395309 7702 tablet_replica.cc:442] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:24.395794 7695 ts_tablet_manager.cc:1428] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:24.395931 7697 raft_consensus.cc:348] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.396190 7697 raft_consensus.cc:374] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:24.396287 7697 raft_consensus.cc:729] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea87e29a4b134c0c9141da03dec72d00, State: Initialized, Role: FOLLOWER
I20260923 20:15:24.396615 7697 consensus_queue.cc:260] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.397286 7697 ts_tablet_manager.cc:1428] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:15:24.397289 7701 tablet_replica.cc:439] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } }
I20260923 20:15:24.398175 7701 tablet_replica.cc:442] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:24.419976 7701 raft_consensus.cc:482] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:24.420271 7701 raft_consensus.cc:504] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.421841 7701 leader_election.cc:288] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103), 1eb80d1e42034d02b0a7873c2e2b8405 (127.5.83.67:38031)
I20260923 20:15:24.428323 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.422135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:58333 (local address 127.5.83.65:34103)
0923 20:15:24.422339 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:24.422348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:24.422862 (+ 514us) server_negotiation.cc:407] Connection header received
0923 20:15:24.423200 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:24.423208 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:24.423285 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:24.423337 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:24.423959 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.424837 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.426624 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.427354 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.427501 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.427626 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:24.427904 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 20:15:24.428026 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 20:15:24.428485 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.422349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:42763 (local address 127.5.83.67:38031)
0923 20:15:24.423070 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:24.423088 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:24.423420 (+ 332us) server_negotiation.cc:407] Connection header received
0923 20:15:24.423622 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:24.423627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:24.423693 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:24.423796 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:24.424378 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.425112 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.426760 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.427511 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.427661 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.427770 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:24.428042 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 20:15:24.428181 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":453,"threads_started":1}
I20260923 20:15:24.428733 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.422134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:34103 (local address 127.5.83.66:58333)
0923 20:15:24.422739 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:24.422764 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:24.423001 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:24.423493 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:24.423496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:24.423516 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:24.423787 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.423794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.424976 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.424982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.426453 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.426465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.427564 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.427570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.427737 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.427769 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:15:24.427877 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:15:24.428004 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":3,"thread_start_us":288,"threads_started":1}
I20260923 20:15:24.428779 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.422138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:38031 (local address 127.5.83.66:42763)
0923 20:15:24.423322 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:24.423336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:24.423480 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:24.423827 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:24.423831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:24.423855 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:24.424192 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.424200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.425251 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.425257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.426615 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.426626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.427727 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.427733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.427889 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.427915 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 20:15:24.428018 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 20:15:24.428150 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"mutex_wait_us":229,"thread_start_us":639,"threads_started":1}
I20260923 20:15:24.429443 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8afec888c25441cabec8b5bb115f700" candidate_uuid: "ea87e29a4b134c0c9141da03dec72d00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf4cfe3807442ec90541fca3b8d471d" is_pre_election: true
I20260923 20:15:24.429616 7630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8afec888c25441cabec8b5bb115f700" candidate_uuid: "ea87e29a4b134c0c9141da03dec72d00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" is_pre_election: true
I20260923 20:15:24.429771 7352 raft_consensus.cc:2454] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea87e29a4b134c0c9141da03dec72d00 in term 0.
I20260923 20:15:24.430027 7630 raft_consensus.cc:2454] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea87e29a4b134c0c9141da03dec72d00 in term 0.
I20260923 20:15:24.430316 7418 leader_election.cc:302] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adf4cfe3807442ec90541fca3b8d471d, ea87e29a4b134c0c9141da03dec72d00; no voters:
I20260923 20:15:24.430737 7701 raft_consensus.cc:2781] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:15:24.430871 7701 raft_consensus.cc:482] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:15:24.430966 7701 raft_consensus.cc:3037] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:24.433151 7701 raft_consensus.cc:504] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.433724 7701 leader_election.cc:288] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [CANDIDATE]: Term 1 election: Requested vote from peers adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103), 1eb80d1e42034d02b0a7873c2e2b8405 (127.5.83.67:38031)
I20260923 20:15:24.434145 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8afec888c25441cabec8b5bb115f700" candidate_uuid: "ea87e29a4b134c0c9141da03dec72d00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf4cfe3807442ec90541fca3b8d471d"
I20260923 20:15:24.434239 7630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8afec888c25441cabec8b5bb115f700" candidate_uuid: "ea87e29a4b134c0c9141da03dec72d00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb80d1e42034d02b0a7873c2e2b8405"
I20260923 20:15:24.434363 7352 raft_consensus.cc:3037] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:24.434465 7630 raft_consensus.cc:3037] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:24.436406 7352 raft_consensus.cc:2454] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea87e29a4b134c0c9141da03dec72d00 in term 1.
I20260923 20:15:24.436779 7630 raft_consensus.cc:2454] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea87e29a4b134c0c9141da03dec72d00 in term 1.
I20260923 20:15:24.436887 7418 leader_election.cc:302] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adf4cfe3807442ec90541fca3b8d471d, ea87e29a4b134c0c9141da03dec72d00; no voters:
I20260923 20:15:24.437203 7701 raft_consensus.cc:2781] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:24.437669 7701 raft_consensus.cc:686] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [term 1 LEADER]: Becoming Leader. State: Replica: ea87e29a4b134c0c9141da03dec72d00, State: Running, Role: LEADER
I20260923 20:15:24.438215 7701 consensus_queue.cc:237] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.441428 7719 tablet_replica.cc:439] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: TxnStatusTablet state changed. Reason: New leader ea87e29a4b134c0c9141da03dec72d00. Latest consensus state: current_term: 1 leader_uuid: "ea87e29a4b134c0c9141da03dec72d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } }
I20260923 20:15:24.441874 7719 tablet_replica.cc:442] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:15:24.442631 7721 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 20:15:24.442791 7189 catalog_manager.cc:5495] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66). New cstate: current_term: 1 leader_uuid: "ea87e29a4b134c0c9141da03dec72d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } health_report { overall_health: UNKNOWN } } }
I20260923 20:15:24.443141 7721 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 20:15:24.474316 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.467722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:43953 (local address 127.0.0.1:41188)
0923 20:15:24.467947 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:24.467967 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:24.468125 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:24.468693 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:24.468699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:24.468712 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:15:24.468952 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.468967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.470144 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.470151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.471737 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.471753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.472579 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.472586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.472772 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.472840 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:15:24.473713 (+ 873us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:15:24.473736 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:15:24.473830 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 20:15:24.473989 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 20:15:24.474465 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.467670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.5.83.66:43953)
0923 20:15:24.467891 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:24.467898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:24.468043 (+ 145us) server_negotiation.cc:407] Connection header received
0923 20:15:24.468339 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:24.468344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:24.468445 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:24.468647 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:15:24.469102 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.470007 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.471884 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.472400 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.472501 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.473042 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:15:24.473599 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:15:24.473711 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:24.473858 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 20:15:24.474162 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:15:24.483474 7352 raft_consensus.cc:1261] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Refusing update from remote peer ea87e29a4b134c0c9141da03dec72d00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:24.483472 7630 raft_consensus.cc:1261] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Refusing update from remote peer ea87e29a4b134c0c9141da03dec72d00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:24.484570 7719 consensus_queue.cc:1035] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:24.485005 7701 consensus_queue.cc:1035] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:24.495361 7702 tablet_replica.cc:439] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d: TxnStatusTablet state changed. Reason: New leader ea87e29a4b134c0c9141da03dec72d00. Latest consensus state: current_term: 1 leader_uuid: "ea87e29a4b134c0c9141da03dec72d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } }
I20260923 20:15:24.495678 7702 tablet_replica.cc:442] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:24.497888 7700 tablet_replica.cc:439] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405: TxnStatusTablet state changed. Reason: New leader ea87e29a4b134c0c9141da03dec72d00. Latest consensus state: current_term: 1 leader_uuid: "ea87e29a4b134c0c9141da03dec72d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } }
I20260923 20:15:24.498200 7700 tablet_replica.cc:442] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:24.503253 7724 mvcc.cc:205] Tried to move back new op lower bound from 7332636772272635904 to 7332636772102561792. Current Snapshot: MvccSnapshot[applied={T|T < 7332636772272635904}]
I20260923 20:15:24.511715 7700 tablet_replica.cc:439] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea87e29a4b134c0c9141da03dec72d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } }
I20260923 20:15:24.512029 7700 tablet_replica.cc:442] T b8afec888c25441cabec8b5bb115f700 P 1eb80d1e42034d02b0a7873c2e2b8405: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:24.513367 7702 tablet_replica.cc:439] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea87e29a4b134c0c9141da03dec72d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } }
I20260923 20:15:24.513621 7702 tablet_replica.cc:442] T b8afec888c25441cabec8b5bb115f700 P adf4cfe3807442ec90541fca3b8d471d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:15:24.517264 7701 tablet_replica.cc:439] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea87e29a4b134c0c9141da03dec72d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } }
I20260923 20:15:24.517230 7729 tablet_replica.cc:439] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea87e29a4b134c0c9141da03dec72d00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } } }
I20260923 20:15:24.517566 7701 tablet_replica.cc:442] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:15:24.517627 7729 tablet_replica.cc:442] T b8afec888c25441cabec8b5bb115f700 P ea87e29a4b134c0c9141da03dec72d00: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:15:24.534960 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.525691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40634 (local address 127.5.83.67:38031)
0923 20:15:24.525896 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:24.525903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:24.529129 (+ 3226us) server_negotiation.cc:407] Connection header received
0923 20:15:24.529390 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:24.529394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:24.529482 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:24.529610 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:15:24.530071 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.530862 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.532860 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.533247 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.533337 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.533829 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:15:24.534362 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:15:24.534463 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:24.534601 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 20:15:24.534667 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:15:24.552830 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.546612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:43953 (local address 127.5.83.67:42481)
0923 20:15:24.547329 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:24.547346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:24.547467 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:24.547808 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:24.547811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:24.547821 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:15:24.548113 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.548121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.549588 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.549593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.550715 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.550726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.551713 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.551717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.551828 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.551876 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:15:24.552305 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:15:24.552319 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 20:15:24.552375 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 20:15:24.552468 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":402,"threads_started":1}
I20260923 20:15:24.552826 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.546636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:42481 (local address 127.5.83.66:43953)
0923 20:15:24.546846 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:24.546853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:24.547425 (+ 572us) server_negotiation.cc:407] Connection header received
0923 20:15:24.547586 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:24.547591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:24.547677 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:24.547775 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:15:24.548286 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.549464 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.550861 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.551541 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.551674 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.552093 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:15:24.552213 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:15:24.552306 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:24.552424 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 20:15:24.552502 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:15:24.562964 7491 raft_consensus.cc:1261] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Refusing update from remote peer 1eb80d1e42034d02b0a7873c2e2b8405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:24.563657 7352 raft_consensus.cc:1261] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Refusing update from remote peer 1eb80d1e42034d02b0a7873c2e2b8405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:24.563771 7704 consensus_queue.cc:1035] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:24.564234 7700 consensus_queue.cc:1035] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:24.569371 7725 mvcc.cc:205] Tried to move back new op lower bound from 7332636772602552320 to 7332636771307569152. Current Snapshot: MvccSnapshot[applied={T|T < 7332636772602552320}]
I20260923 20:15:24.601559 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.594631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:52359 (local address 127.5.83.67:38031)
0923 20:15:24.594805 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:24.594813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:24.595226 (+ 413us) server_negotiation.cc:407] Connection header received
0923 20:15:24.595473 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:24.595479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:24.595566 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:24.595651 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:15:24.596384 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.597422 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.599538 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.600126 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.600246 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.600880 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:15:24.600997 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:15:24.601102 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:24.601201 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 20:15:24.601273 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 20:15:24.601941 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:24.594584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:38031 (local address 127.5.83.66:52359)
0923 20:15:24.595132 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:24.595147 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:24.595302 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:24.595806 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:24.595811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:24.595824 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:15:24.596194 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.596205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.597544 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.597549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.599366 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:24.599380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:24.600431 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:24.600437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:24.600648 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:24.600699 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:15:24.601095 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:15:24.601107 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 20:15:24.601188 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 20:15:24.601315 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260923 20:15:24.638690 7189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:24.638978 7189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:24.643085 7189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a: Sending DeleteTablet for 3 replicas of tablet ac83dc68e29b4d4db44c0fd72ef684de
I20260923 20:15:24.644380 7605 tablet_service.cc:1514] Processing DeleteTablet for tablet ac83dc68e29b4d4db44c0fd72ef684de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:24 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:40630
I20260923 20:15:24.644395 7327 tablet_service.cc:1514] Processing DeleteTablet for tablet ac83dc68e29b4d4db44c0fd72ef684de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:24 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:48618
I20260923 20:15:24.646152 7471 tablet_service.cc:1514] Processing DeleteTablet for tablet ac83dc68e29b4d4db44c0fd72ef684de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:24 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:41180
I20260923 20:15:24.649394 7189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:15:24.650157 7189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:15:24.655860 7745 tablet_replica.cc:330] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d: stopping tablet replica
I20260923 20:15:24.656486 7745 raft_consensus.cc:2229] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:24.656992 7745 raft_consensus.cc:2258] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:24.659022 7745 ts_tablet_manager.cc:1905] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:24.662892 7604 tablet_service.cc:1467] Processing CreateTablet for tablet f15473f21b234ce4b7ceb51f9695c32f (DEFAULT_TABLE table=test-table [id=1c040ce139d341a2b447105176262e21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:24.663511 7604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15473f21b234ce4b7ceb51f9695c32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:24.663728 7327 tablet_service.cc:1467] Processing CreateTablet for tablet f15473f21b234ce4b7ceb51f9695c32f (DEFAULT_TABLE table=test-table [id=1c040ce139d341a2b447105176262e21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:24.664173 7327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15473f21b234ce4b7ceb51f9695c32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:24.667140 7745 ts_tablet_manager.cc:1918] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:15:24.667356 7745 log.cc:1196] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/ac83dc68e29b4d4db44c0fd72ef684de
I20260923 20:15:24.668027 7745 ts_tablet_manager.cc:1939] T ac83dc68e29b4d4db44c0fd72ef684de P adf4cfe3807442ec90541fca3b8d471d: Deleting consensus metadata
I20260923 20:15:24.669081 7470 tablet_service.cc:1467] Processing CreateTablet for tablet f15473f21b234ce4b7ceb51f9695c32f (DEFAULT_TABLE table=test-table [id=1c040ce139d341a2b447105176262e21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:24.669662 7470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f15473f21b234ce4b7ceb51f9695c32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:24.670717 7175 catalog_manager.cc:4841] TS adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103): tablet ac83dc68e29b4d4db44c0fd72ef684de (table test-table [id=2c8f2c20f79648cd8f991dedaa07b748]) successfully deleted
I20260923 20:15:24.671041 7695 tablet_bootstrap.cc:492] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d: Bootstrap starting.
I20260923 20:15:24.673050 7695 tablet_bootstrap.cc:654] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:24.676208 7695 tablet_bootstrap.cc:492] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d: No bootstrap required, opened a new log
I20260923 20:15:24.676406 7695 ts_tablet_manager.cc:1397] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 20:15:24.677470 7697 tablet_bootstrap.cc:492] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00: Bootstrap starting.
I20260923 20:15:24.678238 7695 raft_consensus.cc:348] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.678565 7695 raft_consensus.cc:374] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:24.678686 7695 raft_consensus.cc:729] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf4cfe3807442ec90541fca3b8d471d, State: Initialized, Role: FOLLOWER
I20260923 20:15:24.679104 7697 tablet_bootstrap.cc:654] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:24.679071 7695 consensus_queue.cc:260] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.682922 7694 tablet_bootstrap.cc:492] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405: Bootstrap starting.
I20260923 20:15:24.683768 7695 ts_tablet_manager.cc:1428] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260923 20:15:24.684760 7694 tablet_bootstrap.cc:654] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:24.685241 7746 tablet_replica.cc:330] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00: stopping tablet replica
I20260923 20:15:24.685719 7746 raft_consensus.cc:2229] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:24.686044 7746 raft_consensus.cc:2258] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:24.689061 7746 ts_tablet_manager.cc:1905] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:24.690450 7694 tablet_bootstrap.cc:492] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405: No bootstrap required, opened a new log
I20260923 20:15:24.690630 7694 ts_tablet_manager.cc:1397] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260923 20:15:24.691470 7694 raft_consensus.cc:348] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.691732 7694 raft_consensus.cc:374] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:24.691816 7694 raft_consensus.cc:729] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb80d1e42034d02b0a7873c2e2b8405, State: Initialized, Role: FOLLOWER
I20260923 20:15:24.692118 7694 consensus_queue.cc:260] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.694252 7697 tablet_bootstrap.cc:492] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00: No bootstrap required, opened a new log
I20260923 20:15:24.694470 7697 ts_tablet_manager.cc:1397] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20260923 20:15:24.695140 7694 ts_tablet_manager.cc:1428] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260923 20:15:24.695338 7697 raft_consensus.cc:348] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.695667 7697 raft_consensus.cc:374] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:24.695798 7697 raft_consensus.cc:729] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea87e29a4b134c0c9141da03dec72d00, State: Initialized, Role: FOLLOWER
I20260923 20:15:24.696141 7697 consensus_queue.cc:260] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.698289 7744 tablet_replica.cc:330] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405: stopping tablet replica
I20260923 20:15:24.698762 7744 raft_consensus.cc:2229] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:15:24.698940 7697 ts_tablet_manager.cc:1428] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260923 20:15:24.699445 7744 raft_consensus.cc:2258] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:24.707765 7746 ts_tablet_manager.cc:1918] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:15:24.707948 7746 log.cc:1196] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/ac83dc68e29b4d4db44c0fd72ef684de
I20260923 20:15:24.708544 7746 ts_tablet_manager.cc:1939] T ac83dc68e29b4d4db44c0fd72ef684de P ea87e29a4b134c0c9141da03dec72d00: Deleting consensus metadata
I20260923 20:15:24.710409 7176 catalog_manager.cc:4841] TS ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953): tablet ac83dc68e29b4d4db44c0fd72ef684de (table test-table [id=2c8f2c20f79648cd8f991dedaa07b748]) successfully deleted
I20260923 20:15:24.711107 7744 ts_tablet_manager.cc:1905] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:24.720955 7744 ts_tablet_manager.cc:1918] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:15:24.721264 7744 log.cc:1196] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/ac83dc68e29b4d4db44c0fd72ef684de
I20260923 20:15:24.721874 7744 ts_tablet_manager.cc:1939] T ac83dc68e29b4d4db44c0fd72ef684de P 1eb80d1e42034d02b0a7873c2e2b8405: Deleting consensus metadata
I20260923 20:15:24.723649 7174 catalog_manager.cc:4841] TS 1eb80d1e42034d02b0a7873c2e2b8405 (127.5.83.67:38031): tablet ac83dc68e29b4d4db44c0fd72ef684de (table test-table [id=2c8f2c20f79648cd8f991dedaa07b748]) successfully deleted
I20260923 20:15:24.850708 7704 raft_consensus.cc:482] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:24.850983 7704 raft_consensus.cc:504] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.851835 7704 leader_election.cc:288] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103), ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953)
I20260923 20:15:24.852589 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15473f21b234ce4b7ceb51f9695c32f" candidate_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea87e29a4b134c0c9141da03dec72d00" is_pre_election: true
I20260923 20:15:24.852530 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15473f21b234ce4b7ceb51f9695c32f" candidate_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf4cfe3807442ec90541fca3b8d471d" is_pre_election: true
I20260923 20:15:24.852854 7491 raft_consensus.cc:2454] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb80d1e42034d02b0a7873c2e2b8405 in term 0.
I20260923 20:15:24.852873 7351 raft_consensus.cc:2454] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb80d1e42034d02b0a7873c2e2b8405 in term 0.
I20260923 20:15:24.853317 7558 leader_election.cc:302] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb80d1e42034d02b0a7873c2e2b8405, ea87e29a4b134c0c9141da03dec72d00; no voters:
I20260923 20:15:24.853621 7704 raft_consensus.cc:2781] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:15:24.853717 7704 raft_consensus.cc:482] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:15:24.853838 7704 raft_consensus.cc:3037] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:24.857064 7704 raft_consensus.cc:504] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.858171 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15473f21b234ce4b7ceb51f9695c32f" candidate_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf4cfe3807442ec90541fca3b8d471d"
I20260923 20:15:24.858183 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f15473f21b234ce4b7ceb51f9695c32f" candidate_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea87e29a4b134c0c9141da03dec72d00"
I20260923 20:15:24.858368 7351 raft_consensus.cc:3037] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:24.858389 7491 raft_consensus.cc:3037] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:24.861509 7351 raft_consensus.cc:2454] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb80d1e42034d02b0a7873c2e2b8405 in term 1.
I20260923 20:15:24.861728 7491 raft_consensus.cc:2454] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb80d1e42034d02b0a7873c2e2b8405 in term 1.
I20260923 20:15:24.862100 7557 leader_election.cc:302] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb80d1e42034d02b0a7873c2e2b8405, adf4cfe3807442ec90541fca3b8d471d; no voters:
I20260923 20:15:24.862341 7704 leader_election.cc:288] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [CANDIDATE]: Term 1 election: Requested vote from peers adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103), ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953)
I20260923 20:15:24.862464 7700 raft_consensus.cc:2781] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:24.862648 7700 raft_consensus.cc:686] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 LEADER]: Becoming Leader. State: Replica: 1eb80d1e42034d02b0a7873c2e2b8405, State: Running, Role: LEADER
I20260923 20:15:24.862960 7700 consensus_queue.cc:237] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:24.868702 7188 catalog_manager.cc:5495] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb80d1e42034d02b0a7873c2e2b8405 (127.5.83.67). New cstate: current_term: 1 leader_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } health_report { overall_health: HEALTHY } } }
I20260923 20:15:24.937623 7491 raft_consensus.cc:1261] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Refusing update from remote peer 1eb80d1e42034d02b0a7873c2e2b8405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:24.937845 7351 raft_consensus.cc:1261] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Refusing update from remote peer 1eb80d1e42034d02b0a7873c2e2b8405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:24.938354 7704 consensus_queue.cc:1035] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:24.938656 7700 consensus_queue.cc:1035] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:15:24.962284 7725 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 20:15:26.549642 7188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:26.549963 7188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:26.554021 7188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a: Sending DeleteTablet for 3 replicas of tablet f15473f21b234ce4b7ceb51f9695c32f
I20260923 20:15:26.554852 7327 tablet_service.cc:1514] Processing DeleteTablet for tablet f15473f21b234ce4b7ceb51f9695c32f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:26 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:48618
I20260923 20:15:26.554916 7471 tablet_service.cc:1514] Processing DeleteTablet for tablet f15473f21b234ce4b7ceb51f9695c32f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:26 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:41180
I20260923 20:15:26.555011 7605 tablet_service.cc:1514] Processing DeleteTablet for tablet f15473f21b234ce4b7ceb51f9695c32f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:26 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:40630
I20260923 20:15:26.565129 7777 tablet_replica.cc:330] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00: stopping tablet replica
I20260923 20:15:26.565584 7777 raft_consensus.cc:2229] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:26.565990 7777 raft_consensus.cc:2258] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:26.567142 7778 tablet_replica.cc:330] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405: stopping tablet replica
I20260923 20:15:26.567288 7776 tablet_replica.cc:330] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d: stopping tablet replica
I20260923 20:15:26.567299 7777 ts_tablet_manager.cc:1905] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:26.567387 7778 raft_consensus.cc:2229] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:15:26.567564 7776 raft_consensus.cc:2229] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:26.567814 7778 raft_consensus.cc:2258] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:26.567907 7776 raft_consensus.cc:2258] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:26.568836 7778 ts_tablet_manager.cc:1905] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:26.569085 7776 ts_tablet_manager.cc:1905] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:26.574158 7777 ts_tablet_manager.cc:1918] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:15:26.574369 7777 log.cc:1196] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/f15473f21b234ce4b7ceb51f9695c32f
I20260923 20:15:26.574884 7777 ts_tablet_manager.cc:1939] T f15473f21b234ce4b7ceb51f9695c32f P ea87e29a4b134c0c9141da03dec72d00: Deleting consensus metadata
I20260923 20:15:26.575587 7778 ts_tablet_manager.cc:1918] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:15:26.575774 7778 log.cc:1196] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/f15473f21b234ce4b7ceb51f9695c32f
I20260923 20:15:26.576184 7176 catalog_manager.cc:4841] TS ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953): tablet f15473f21b234ce4b7ceb51f9695c32f (table test-table [id=1c040ce139d341a2b447105176262e21]) successfully deleted
I20260923 20:15:26.576284 7778 ts_tablet_manager.cc:1939] T f15473f21b234ce4b7ceb51f9695c32f P 1eb80d1e42034d02b0a7873c2e2b8405: Deleting consensus metadata
I20260923 20:15:26.576453 7776 ts_tablet_manager.cc:1918] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:15:26.576658 7776 log.cc:1196] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/f15473f21b234ce4b7ceb51f9695c32f
I20260923 20:15:26.577149 7776 ts_tablet_manager.cc:1939] T f15473f21b234ce4b7ceb51f9695c32f P adf4cfe3807442ec90541fca3b8d471d: Deleting consensus metadata
I20260923 20:15:26.577358 7174 catalog_manager.cc:4841] TS 1eb80d1e42034d02b0a7873c2e2b8405 (127.5.83.67:38031): tablet f15473f21b234ce4b7ceb51f9695c32f (table test-table [id=1c040ce139d341a2b447105176262e21]) successfully deleted
I20260923 20:15:26.578315 7175 catalog_manager.cc:4841] TS adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103): tablet f15473f21b234ce4b7ceb51f9695c32f (table test-table [id=1c040ce139d341a2b447105176262e21]) successfully deleted
Sep 23 20:15:26 dist-test-slave-vl24 krb5kdc[7145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:15:26 dist-test-slave-vl24 krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:26.611241 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:26.596836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.5.83.126:40277)
0923 20:15:26.597488 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:26.597495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:26.597630 (+ 135us) server_negotiation.cc:407] Connection header received
0923 20:15:26.597862 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:26.597867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:26.597942 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:26.598058 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:26.599386 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.600173 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.601494 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.601899 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.602034 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:26.606050 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:26.606055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:26.606058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:26.606094 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:26.608687 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:26.609347 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:26.609352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:26.609356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:26.609478 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:26.609849 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:26.609854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:26.609857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:26.610127 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:26.610375 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:26.610677 (+ 302us) server_negotiation.cc:299] Negotiation successful
0923 20:15:26.610905 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":313,"threads_started":1}
I20260923 20:15:26.614636 7188 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:15:26.615252 7188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:15:26.621443 7327 tablet_service.cc:1467] Processing CreateTablet for tablet 857ca0a8d8144790a8856408bf6923ae (DEFAULT_TABLE table=test-table [id=81f5fe8fef494dd5b5be9185606d5b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:26.621658 7471 tablet_service.cc:1467] Processing CreateTablet for tablet 857ca0a8d8144790a8856408bf6923ae (DEFAULT_TABLE table=test-table [id=81f5fe8fef494dd5b5be9185606d5b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:26.621905 7327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857ca0a8d8144790a8856408bf6923ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:26.621905 7605 tablet_service.cc:1467] Processing CreateTablet for tablet 857ca0a8d8144790a8856408bf6923ae (DEFAULT_TABLE table=test-table [id=81f5fe8fef494dd5b5be9185606d5b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:26.622037 7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857ca0a8d8144790a8856408bf6923ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:26.622288 7605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857ca0a8d8144790a8856408bf6923ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:26.627414 7787 tablet_bootstrap.cc:492] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00: Bootstrap starting.
I20260923 20:15:26.629128 7787 tablet_bootstrap.cc:654] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:26.631345 7787 tablet_bootstrap.cc:492] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00: No bootstrap required, opened a new log
I20260923 20:15:26.631460 7787 ts_tablet_manager.cc:1397] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 20:15:26.631788 7789 tablet_bootstrap.cc:492] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405: Bootstrap starting.
I20260923 20:15:26.632247 7787 raft_consensus.cc:348] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.632508 7787 raft_consensus.cc:374] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:26.632634 7787 raft_consensus.cc:729] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea87e29a4b134c0c9141da03dec72d00, State: Initialized, Role: FOLLOWER
I20260923 20:15:26.632867 7787 consensus_queue.cc:260] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.633407 7787 ts_tablet_manager.cc:1428] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260923 20:15:26.633536 7789 tablet_bootstrap.cc:654] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:26.633857 7788 tablet_bootstrap.cc:492] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d: Bootstrap starting.
I20260923 20:15:26.635653 7788 tablet_bootstrap.cc:654] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:26.635703 7789 tablet_bootstrap.cc:492] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405: No bootstrap required, opened a new log
I20260923 20:15:26.635890 7789 ts_tablet_manager.cc:1397] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260923 20:15:26.636685 7789 raft_consensus.cc:348] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.637019 7789 raft_consensus.cc:374] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:26.637181 7789 raft_consensus.cc:729] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb80d1e42034d02b0a7873c2e2b8405, State: Initialized, Role: FOLLOWER
I20260923 20:15:26.637590 7789 consensus_queue.cc:260] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.638167 7789 ts_tablet_manager.cc:1428] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:15:26.639823 7788 tablet_bootstrap.cc:492] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d: No bootstrap required, opened a new log
I20260923 20:15:26.639988 7788 ts_tablet_manager.cc:1397] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260923 20:15:26.640944 7788 raft_consensus.cc:348] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.641219 7788 raft_consensus.cc:374] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:26.641343 7788 raft_consensus.cc:729] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf4cfe3807442ec90541fca3b8d471d, State: Initialized, Role: FOLLOWER
I20260923 20:15:26.641588 7788 consensus_queue.cc:260] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.642465 7788 ts_tablet_manager.cc:1428] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 20:15:26.686952 7793 raft_consensus.cc:482] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:26.687225 7793 raft_consensus.cc:504] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.689025 7793 leader_election.cc:288] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953), 1eb80d1e42034d02b0a7873c2e2b8405 (127.5.83.67:38031)
I20260923 20:15:26.698896 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:26.689748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:43975 (local address 127.5.83.66:43953)
0923 20:15:26.691361 (+ 1613us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:26.691365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:26.691385 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:15:26.691443 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:26.691447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:26.691506 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:26.691628 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:26.692212 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.693373 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.696262 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.697572 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.697730 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:26.697838 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:26.698416 (+ 578us) server_negotiation.cc:299] Negotiation successful
0923 20:15:26.698556 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1468,"thread_start_us":1309,"threads_started":1}
I20260923 20:15:26.698872 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:26.690000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:41971 (local address 127.5.83.67:38031)
0923 20:15:26.691645 (+ 1645us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:26.691649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:26.691671 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:15:26.691736 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:26.691739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:26.691793 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:26.691876 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:26.692583 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.693638 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.696265 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.697579 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.697718 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:26.697849 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:26.698417 (+ 568us) server_negotiation.cc:299] Negotiation successful
0923 20:15:26.698556 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1542,"thread_start_us":1395,"threads_started":1}
I20260923 20:15:26.699018 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:26.689278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:38031 (local address 127.5.83.65:41971)
0923 20:15:26.690024 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:26.690050 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:26.690229 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:26.691926 (+ 1697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:26.691931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:26.691959 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:26.692369 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:26.692378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.693749 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.693754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:26.696084 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:26.696100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.697924 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.697932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:26.698136 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:26.698260 (+ 124us) client_negotiation.cc:769] Sending connection context
0923 20:15:26.698369 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 20:15:26.698540 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":133,"thread_start_us":621,"threads_started":1}
I20260923 20:15:26.699533 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:26.689339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:43953 (local address 127.5.83.65:43975)
0923 20:15:26.691059 (+ 1720us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:26.691092 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:26.691212 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:26.691643 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:26.691648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:26.691671 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:26.692008 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:26.692025 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.693520 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.693523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:26.696089 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:26.696100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.697933 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.697939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:26.698200 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:26.698257 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 20:15:26.698373 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 20:15:26.698540 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1567,"mutex_wait_us":147,"thread_start_us":1379,"threads_started":1}
I20260923 20:15:26.699694 7630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857ca0a8d8144790a8856408bf6923ae" candidate_uuid: "adf4cfe3807442ec90541fca3b8d471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" is_pre_election: true
I20260923 20:15:26.699958 7630 raft_consensus.cc:2454] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adf4cfe3807442ec90541fca3b8d471d in term 0.
I20260923 20:15:26.700402 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857ca0a8d8144790a8856408bf6923ae" candidate_uuid: "adf4cfe3807442ec90541fca3b8d471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea87e29a4b134c0c9141da03dec72d00" is_pre_election: true
I20260923 20:15:26.700664 7491 raft_consensus.cc:2454] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adf4cfe3807442ec90541fca3b8d471d in term 0.
I20260923 20:15:26.700592 7277 leader_election.cc:302] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb80d1e42034d02b0a7873c2e2b8405, adf4cfe3807442ec90541fca3b8d471d; no voters:
I20260923 20:15:26.701139 7793 raft_consensus.cc:2781] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:15:26.701272 7793 raft_consensus.cc:482] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:15:26.701371 7793 raft_consensus.cc:3037] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:26.703576 7793 raft_consensus.cc:504] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.704208 7793 leader_election.cc:288] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [CANDIDATE]: Term 1 election: Requested vote from peers ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953), 1eb80d1e42034d02b0a7873c2e2b8405 (127.5.83.67:38031)
I20260923 20:15:26.704677 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857ca0a8d8144790a8856408bf6923ae" candidate_uuid: "adf4cfe3807442ec90541fca3b8d471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea87e29a4b134c0c9141da03dec72d00"
I20260923 20:15:26.704811 7630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857ca0a8d8144790a8856408bf6923ae" candidate_uuid: "adf4cfe3807442ec90541fca3b8d471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb80d1e42034d02b0a7873c2e2b8405"
I20260923 20:15:26.704885 7491 raft_consensus.cc:3037] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:26.705010 7630 raft_consensus.cc:3037] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:26.707861 7630 raft_consensus.cc:2454] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adf4cfe3807442ec90541fca3b8d471d in term 1.
I20260923 20:15:26.707860 7491 raft_consensus.cc:2454] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adf4cfe3807442ec90541fca3b8d471d in term 1.
I20260923 20:15:26.708242 7279 leader_election.cc:302] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adf4cfe3807442ec90541fca3b8d471d, ea87e29a4b134c0c9141da03dec72d00; no voters:
I20260923 20:15:26.708619 7793 raft_consensus.cc:2781] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:26.709044 7793 raft_consensus.cc:686] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 1 LEADER]: Becoming Leader. State: Replica: adf4cfe3807442ec90541fca3b8d471d, State: Running, Role: LEADER
I20260923 20:15:26.709451 7793 consensus_queue.cc:237] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.712903 7188 catalog_manager.cc:5495] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d reported cstate change: term changed from 0 to 1, leader changed from <none> to adf4cfe3807442ec90541fca3b8d471d (127.5.83.65). New cstate: current_term: 1 leader_uuid: "adf4cfe3807442ec90541fca3b8d471d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } health_report { overall_health: UNKNOWN } } }
I20260923 20:15:26.745779 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:26.739534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48622 (local address 127.5.83.65:34103)
0923 20:15:26.740104 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:26.740109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:26.740137 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:15:26.740226 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:26.740229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:26.740305 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:26.740478 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:15:26.740994 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.741807 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.743395 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.743880 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.743980 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:26.744422 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:15:26.745124 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:15:26.745233 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:26.745345 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 20:15:26.745410 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":303,"threads_started":1}
I20260923 20:15:26.761281 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:26.755832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:36821 (local address 127.5.83.66:43953)
0923 20:15:26.756060 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:26.756069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:26.756593 (+ 524us) server_negotiation.cc:407] Connection header received
0923 20:15:26.756827 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:26.756831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:26.756896 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:26.756993 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:15:26.757467 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.758166 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.759502 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.759927 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.760073 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:26.760485 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:15:26.760648 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:15:26.760748 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:26.760897 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 20:15:26.760974 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 20:15:26.761384 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:26.755782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:43953 (local address 127.5.83.65:36821)
0923 20:15:26.756457 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:26.756474 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:26.756669 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:26.757007 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:26.757011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:26.757023 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:15:26.757324 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:26.757332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.758299 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.758306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:26.759365 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:26.759375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.760107 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.760114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:26.760258 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:26.760313 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:15:26.760763 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:15:26.760780 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 20:15:26.760862 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 20:15:26.760998 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":297,"threads_started":1}
I20260923 20:15:26.770742 7630 raft_consensus.cc:1261] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Refusing update from remote peer adf4cfe3807442ec90541fca3b8d471d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:26.770741 7491 raft_consensus.cc:1261] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Refusing update from remote peer adf4cfe3807442ec90541fca3b8d471d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:26.771436 7793 consensus_queue.cc:1035] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:26.771725 7798 consensus_queue.cc:1035] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:26.803469 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:26.797937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:35561 (local address 127.5.83.65:34103)
0923 20:15:26.798138 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:26.798144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:26.798687 (+ 543us) server_negotiation.cc:407] Connection header received
0923 20:15:26.798820 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:26.798824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:26.798883 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:26.798953 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:15:26.799481 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.800229 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.801732 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.802408 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.802529 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:26.802858 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:15:26.802952 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:15:26.803049 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:26.803190 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 20:15:26.803250 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:15:26.803591 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:26.797919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:34103 (local address 127.5.83.66:35561)
0923 20:15:26.798601 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:26.798617 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:26.798724 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:26.798980 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:26.798983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:26.798992 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:15:26.799195 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:26.799203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.800377 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.800381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:26.801601 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:26.801611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:26.802554 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:26.802558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:26.802689 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:26.802724 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:15:26.803055 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:15:26.803067 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 20:15:26.803144 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 20:15:26.803250 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":410,"threads_started":1}
I20260923 20:15:26.840860 7189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:26.841130 7189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:26.844146 7189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a: Sending DeleteTablet for 3 replicas of tablet 857ca0a8d8144790a8856408bf6923ae
I20260923 20:15:26.844951 7327 tablet_service.cc:1514] Processing DeleteTablet for tablet 857ca0a8d8144790a8856408bf6923ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:26 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:48618
I20260923 20:15:26.845293 7776 tablet_replica.cc:330] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d: stopping tablet replica
I20260923 20:15:26.845378 7605 tablet_service.cc:1514] Processing DeleteTablet for tablet 857ca0a8d8144790a8856408bf6923ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:26 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:40630
I20260923 20:15:26.845506 7776 raft_consensus.cc:2229] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:15:26.845643 7778 tablet_replica.cc:330] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405: stopping tablet replica
I20260923 20:15:26.845837 7778 raft_consensus.cc:2229] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:26.846026 7778 raft_consensus.cc:2258] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:26.846012 7776 raft_consensus.cc:2258] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:26.846276 7189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 20:15:26.846890 7471 tablet_service.cc:1514] Processing DeleteTablet for tablet 857ca0a8d8144790a8856408bf6923ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:26 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:41180
W20260923 20:15:26.846961 7189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:15:26.847328 7777 tablet_replica.cc:330] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00: stopping tablet replica
I20260923 20:15:26.847536 7777 raft_consensus.cc:2229] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:26.847671 7778 ts_tablet_manager.cc:1905] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:26.847752 7777 raft_consensus.cc:2258] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:26.847826 7776 ts_tablet_manager.cc:1905] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:26.849675 7777 ts_tablet_manager.cc:1905] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:26.853902 7776 ts_tablet_manager.cc:1918] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:15:26.854107 7776 log.cc:1196] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/857ca0a8d8144790a8856408bf6923ae
I20260923 20:15:26.854161 7778 ts_tablet_manager.cc:1918] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:15:26.854348 7778 log.cc:1196] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/857ca0a8d8144790a8856408bf6923ae
I20260923 20:15:26.854637 7776 ts_tablet_manager.cc:1939] T 857ca0a8d8144790a8856408bf6923ae P adf4cfe3807442ec90541fca3b8d471d: Deleting consensus metadata
I20260923 20:15:26.854777 7778 ts_tablet_manager.cc:1939] T 857ca0a8d8144790a8856408bf6923ae P 1eb80d1e42034d02b0a7873c2e2b8405: Deleting consensus metadata
I20260923 20:15:26.856056 7175 catalog_manager.cc:4841] TS adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103): tablet 857ca0a8d8144790a8856408bf6923ae (table test-table [id=81f5fe8fef494dd5b5be9185606d5b92]) successfully deleted
I20260923 20:15:26.856422 7471 tablet_service.cc:1467] Processing CreateTablet for tablet 0baf3b99fa8a4246a45631b67ceb19f4 (DEFAULT_TABLE table=test-table [id=e4b6ed27aa4f4940b3bb0d778e5946d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:26.856506 7327 tablet_service.cc:1467] Processing CreateTablet for tablet 0baf3b99fa8a4246a45631b67ceb19f4 (DEFAULT_TABLE table=test-table [id=e4b6ed27aa4f4940b3bb0d778e5946d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:26.856649 7174 catalog_manager.cc:4841] TS 1eb80d1e42034d02b0a7873c2e2b8405 (127.5.83.67:38031): tablet 857ca0a8d8144790a8856408bf6923ae (table test-table [id=81f5fe8fef494dd5b5be9185606d5b92]) successfully deleted
I20260923 20:15:26.856976 7327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0baf3b99fa8a4246a45631b67ceb19f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:26.857007 7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0baf3b99fa8a4246a45631b67ceb19f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:26.857473 7605 tablet_service.cc:1467] Processing CreateTablet for tablet 0baf3b99fa8a4246a45631b67ceb19f4 (DEFAULT_TABLE table=test-table [id=e4b6ed27aa4f4940b3bb0d778e5946d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:26.857849 7605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0baf3b99fa8a4246a45631b67ceb19f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:26.861770 7787 tablet_bootstrap.cc:492] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00: Bootstrap starting.
I20260923 20:15:26.863204 7788 tablet_bootstrap.cc:492] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d: Bootstrap starting.
I20260923 20:15:26.863420 7787 tablet_bootstrap.cc:654] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:26.865149 7788 tablet_bootstrap.cc:654] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:26.866406 7787 tablet_bootstrap.cc:492] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00: No bootstrap required, opened a new log
I20260923 20:15:26.866637 7787 ts_tablet_manager.cc:1397] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260923 20:15:26.867471 7787 raft_consensus.cc:348] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.867802 7787 raft_consensus.cc:374] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:26.867918 7787 raft_consensus.cc:729] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea87e29a4b134c0c9141da03dec72d00, State: Initialized, Role: FOLLOWER
I20260923 20:15:26.868242 7787 consensus_queue.cc:260] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.869274 7777 ts_tablet_manager.cc:1918] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:15:26.869678 7789 tablet_bootstrap.cc:492] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405: Bootstrap starting.
I20260923 20:15:26.871783 7789 tablet_bootstrap.cc:654] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:26.874029 7777 log.cc:1196] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/857ca0a8d8144790a8856408bf6923ae
I20260923 20:15:26.874565 7777 ts_tablet_manager.cc:1939] T 857ca0a8d8144790a8856408bf6923ae P ea87e29a4b134c0c9141da03dec72d00: Deleting consensus metadata
I20260923 20:15:26.870081 7787 ts_tablet_manager.cc:1428] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260923 20:15:26.875978 7789 tablet_bootstrap.cc:492] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405: No bootstrap required, opened a new log
I20260923 20:15:26.876163 7789 ts_tablet_manager.cc:1397] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 20:15:26.877041 7789 raft_consensus.cc:348] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.877308 7789 raft_consensus.cc:374] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:26.877398 7789 raft_consensus.cc:729] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb80d1e42034d02b0a7873c2e2b8405, State: Initialized, Role: FOLLOWER
I20260923 20:15:26.877688 7789 consensus_queue.cc:260] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.878295 7789 ts_tablet_manager.cc:1428] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 20:15:26.880255 7176 catalog_manager.cc:4841] TS ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953): tablet 857ca0a8d8144790a8856408bf6923ae (table test-table [id=81f5fe8fef494dd5b5be9185606d5b92]) successfully deleted
I20260923 20:15:26.881269 7788 tablet_bootstrap.cc:492] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d: No bootstrap required, opened a new log
I20260923 20:15:26.881472 7788 ts_tablet_manager.cc:1397] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260923 20:15:26.882350 7788 raft_consensus.cc:348] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.882622 7788 raft_consensus.cc:374] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:26.882725 7788 raft_consensus.cc:729] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf4cfe3807442ec90541fca3b8d471d, State: Initialized, Role: FOLLOWER
I20260923 20:15:26.883071 7788 consensus_queue.cc:260] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.883819 7788 ts_tablet_manager.cc:1428] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 20:15:26.974128 7766 raft_consensus.cc:482] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:26.974325 7766 raft_consensus.cc:504] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.975106 7766 leader_election.cc:288] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103), ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953)
I20260923 20:15:26.975689 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0baf3b99fa8a4246a45631b67ceb19f4" candidate_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf4cfe3807442ec90541fca3b8d471d" is_pre_election: true
I20260923 20:15:26.975767 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0baf3b99fa8a4246a45631b67ceb19f4" candidate_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea87e29a4b134c0c9141da03dec72d00" is_pre_election: true
I20260923 20:15:26.975970 7352 raft_consensus.cc:2454] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb80d1e42034d02b0a7873c2e2b8405 in term 0.
I20260923 20:15:26.975989 7491 raft_consensus.cc:2454] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb80d1e42034d02b0a7873c2e2b8405 in term 0.
I20260923 20:15:26.976447 7558 leader_election.cc:302] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb80d1e42034d02b0a7873c2e2b8405, ea87e29a4b134c0c9141da03dec72d00; no voters:
I20260923 20:15:26.976766 7766 raft_consensus.cc:2781] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:15:26.976907 7766 raft_consensus.cc:482] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:15:26.976999 7766 raft_consensus.cc:3037] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:26.980147 7766 raft_consensus.cc:504] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.981031 7766 leader_election.cc:288] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [CANDIDATE]: Term 1 election: Requested vote from peers adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103), ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953)
I20260923 20:15:26.981344 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0baf3b99fa8a4246a45631b67ceb19f4" candidate_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea87e29a4b134c0c9141da03dec72d00"
I20260923 20:15:26.981350 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0baf3b99fa8a4246a45631b67ceb19f4" candidate_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf4cfe3807442ec90541fca3b8d471d"
I20260923 20:15:26.981597 7352 raft_consensus.cc:3037] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:26.981590 7491 raft_consensus.cc:3037] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:26.984910 7491 raft_consensus.cc:2454] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb80d1e42034d02b0a7873c2e2b8405 in term 1.
I20260923 20:15:26.985044 7352 raft_consensus.cc:2454] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb80d1e42034d02b0a7873c2e2b8405 in term 1.
I20260923 20:15:26.985503 7558 leader_election.cc:302] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb80d1e42034d02b0a7873c2e2b8405, ea87e29a4b134c0c9141da03dec72d00; no voters:
I20260923 20:15:26.985832 7766 raft_consensus.cc:2781] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:26.986017 7766 raft_consensus.cc:686] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 LEADER]: Becoming Leader. State: Replica: 1eb80d1e42034d02b0a7873c2e2b8405, State: Running, Role: LEADER
I20260923 20:15:26.986346 7766 consensus_queue.cc:237] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } }
I20260923 20:15:26.989127 7188 catalog_manager.cc:5495] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb80d1e42034d02b0a7873c2e2b8405 (127.5.83.67). New cstate: current_term: 1 leader_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb80d1e42034d02b0a7873c2e2b8405" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38031 } health_report { overall_health: HEALTHY } } }
I20260923 20:15:27.033658 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:27.028371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.5.83.67:38031)
0923 20:15:27.028567 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:27.028582 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:27.028604 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:15:27.028810 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:27.028816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:27.028901 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:27.029007 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:15:27.029405 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:27.030461 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:27.031760 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:27.032407 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:27.032552 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:27.032938 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:15:27.033062 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:15:27.033153 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:27.033269 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 20:15:27.033349 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:15:27.043454 7352 raft_consensus.cc:1261] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Refusing update from remote peer 1eb80d1e42034d02b0a7873c2e2b8405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:27.043545 7491 raft_consensus.cc:1261] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Refusing update from remote peer 1eb80d1e42034d02b0a7873c2e2b8405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:27.044080 7766 consensus_queue.cc:1035] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adf4cfe3807442ec90541fca3b8d471d" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:27.044401 7771 consensus_queue.cc:1035] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea87e29a4b134c0c9141da03dec72d00" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 43953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:27.056011 7758 mvcc.cc:205] Tried to move back new op lower bound from 7332636782765240320 to 7332636782538403840. Current Snapshot: MvccSnapshot[applied={T|T < 7332636782765240320}]
W20260923 20:15:27.069293 7757 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 20:15:28.683620 7189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:28.683873 7189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:28.687948 7189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3900417b35834f8ebeac761a61d9320a: Sending DeleteTablet for 3 replicas of tablet 0baf3b99fa8a4246a45631b67ceb19f4
I20260923 20:15:28.688724 7327 tablet_service.cc:1514] Processing DeleteTablet for tablet 0baf3b99fa8a4246a45631b67ceb19f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:28 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:48618
I20260923 20:15:28.688762 7470 tablet_service.cc:1514] Processing DeleteTablet for tablet 0baf3b99fa8a4246a45631b67ceb19f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:28 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:41180
I20260923 20:15:28.689177 7605 tablet_service.cc:1514] Processing DeleteTablet for tablet 0baf3b99fa8a4246a45631b67ceb19f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:28 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:40630
I20260923 20:15:28.689667 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 7261
I20260923 20:15:28.699329 7843 tablet_replica.cc:330] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405: stopping tablet replica
I20260923 20:15:28.699805 7843 raft_consensus.cc:2229] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:15:28.700413 7843 raft_consensus.cc:2258] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:28.701546 7841 tablet_replica.cc:330] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d: stopping tablet replica
I20260923 20:15:28.701771 7843 ts_tablet_manager.cc:1905] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:28.701871 7841 raft_consensus.cc:2229] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:28.702133 7841 raft_consensus.cc:2258] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:28.703200 7841 ts_tablet_manager.cc:1905] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:28.708586 7843 ts_tablet_manager.cc:1918] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:15:28.708741 7843 log.cc:1196] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/0baf3b99fa8a4246a45631b67ceb19f4
I20260923 20:15:28.709390 7842 tablet_replica.cc:330] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00: stopping tablet replica
I20260923 20:15:28.709592 7842 raft_consensus.cc:2229] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:28.709827 7842 raft_consensus.cc:2258] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:28.710449 7841 ts_tablet_manager.cc:1918] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:15:28.710578 7841 log.cc:1196] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/0baf3b99fa8a4246a45631b67ceb19f4
I20260923 20:15:28.710723 7842 ts_tablet_manager.cc:1905] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:28.711086 7841 ts_tablet_manager.cc:1939] T 0baf3b99fa8a4246a45631b67ceb19f4 P adf4cfe3807442ec90541fca3b8d471d: Deleting consensus metadata
I20260923 20:15:28.713171 7843 ts_tablet_manager.cc:1939] T 0baf3b99fa8a4246a45631b67ceb19f4 P 1eb80d1e42034d02b0a7873c2e2b8405: Deleting consensus metadata
Sep 23 20:15:28 dist-test-slave-vl24 krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:28.716589 7842 ts_tablet_manager.cc:1918] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 20:15:28.716773 7842 log.cc:1196] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/0baf3b99fa8a4246a45631b67ceb19f4
I20260923 20:15:28.716759 7175 catalog_manager.cc:4841] TS adf4cfe3807442ec90541fca3b8d471d (127.5.83.65:34103): tablet 0baf3b99fa8a4246a45631b67ceb19f4 (table test-table [id=e4b6ed27aa4f4940b3bb0d778e5946d4]) successfully deleted
I20260923 20:15:28.717181 7174 catalog_manager.cc:4841] TS 1eb80d1e42034d02b0a7873c2e2b8405 (127.5.83.67:38031): tablet 0baf3b99fa8a4246a45631b67ceb19f4 (table test-table [id=e4b6ed27aa4f4940b3bb0d778e5946d4]) successfully deleted
I20260923 20:15:28.717240 7842 ts_tablet_manager.cc:1939] T 0baf3b99fa8a4246a45631b67ceb19f4 P ea87e29a4b134c0c9141da03dec72d00: Deleting consensus metadata
I20260923 20:15:28.718219 7176 catalog_manager.cc:4841] TS ea87e29a4b134c0c9141da03dec72d00 (127.5.83.66:43953): tablet 0baf3b99fa8a4246a45631b67ceb19f4 (table test-table [id=e4b6ed27aa4f4940b3bb0d778e5946d4]) successfully deleted
I20260923 20:15:28.725136 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:28.692185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.5.83.65:34103)
0923 20:15:28.697436 (+ 5251us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:28.697446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:28.705158 (+ 7712us) server_negotiation.cc:407] Connection header received
0923 20:15:28.705529 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:28.705537 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:28.705653 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:28.705755 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:28.708163 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:28.709498 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:28.711261 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:28.711973 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:28.712107 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:28.717559 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:28.717579 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:28.717602 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:28.717664 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:28.721847 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:28.722595 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:28.722605 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:28.722609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:28.722810 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:28.723370 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:28.723376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:28.723379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:28.723898 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:28.724224 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:28.724471 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 20:15:28.724761 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5057,"thread_start_us":4894,"threads_started":1}
W20260923 20:15:28.725883 7392 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42338
W20260923 20:15:28.726595 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 7261: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:15:28.726814 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 7261
I20260923 20:15:28.751451 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 7401
Sep 23 20:15:28 dist-test-slave-vl24 krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:15:28.766990 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:28.752327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.5.83.66:43953)
0923 20:15:28.752860 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:28.752864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:28.752880 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:15:28.752933 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:28.752936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:28.752997 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:28.753091 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:28.754432 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:28.755157 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:28.756403 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:28.756934 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:28.757030 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:28.761442 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:28.761457 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:28.761474 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:28.761512 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:28.764152 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:28.765054 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:28.765062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:28.765064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:28.765231 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:28.765587 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:28.765590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:28.765592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:28.765918 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:28.766143 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:28.766536 (+ 393us) server_negotiation.cc:299] Negotiation successful
0923 20:15:28.766717 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":303,"threads_started":1}
W20260923 20:15:28.767882 7531 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51954
W20260923 20:15:28.768610 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 7401: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:15:28.768817 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 7401
I20260923 20:15:28.792151 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 7540
Sep 23 20:15:28 dist-test-slave-vl24 krb5kdc[7145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194526, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:15:28.808480 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:28.792977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.5.83.67:38031)
0923 20:15:28.793584 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:28.793599 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:28.793616 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:15:28.793674 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:28.793678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:28.793747 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:28.793879 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:28.795330 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:28.796104 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:28.797581 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:28.798018 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:28.798165 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:28.802803 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:28.802819 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:28.802836 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:28.802870 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:28.805737 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:28.806390 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:28.806397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:28.806399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:28.806556 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:28.806914 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:28.806918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:28.806920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:28.807332 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:28.807599 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:28.807828 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 20:15:28.808126 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":277,"threads_started":1}
W20260923 20:15:28.809347 7670 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41996
W20260923 20:15:28.810060 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 7540: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:15:28.810225 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 7540
I20260923 20:15:28.833017 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 7158
I20260923 20:15:28.834307 7230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:28.952006 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 7158
2026-09-23T20:15:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8548 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:15:29 dist-test-slave-vl24 krb5kdc[7851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:15:29 dist-test-slave-vl24 krb5kdc[7851](info): set up 2 sockets
Sep 23 20:15:29 dist-test-slave-vl24 krb5kdc[7851](info): commencing operation
krb5kdc: starting...
W20260923 20:15:30.038080 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:15:30 dist-test-slave-vl24 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:15:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:15:30.236173 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36611
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:36611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:30.563441 7864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:30.563771 7864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:30.563930 7864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:30.571255 7864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:30.571430 7864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:30.571548 7864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:30.571655 7864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:30.571738 7864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:15:30.582098 7864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:36611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:30.583678 7864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:30.585037 7864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:30.597573 7869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:30.597555 7870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:30.597522 7872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:30.598470 7864 server_base.cc:1034] running on GCE node
I20260923 20:15:30.599259 7864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:30.600754 7864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:30.601959 7864 hybrid_clock.cc:648] HybridClock initialized: now 1790194530601896 us; error 59 us; skew 500 ppm
I20260923 20:15:30.602346 7864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:30 dist-test-slave-vl24 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194530, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:30.611826 7864 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:15:30.613575 7864 webserver.cc:460] Webserver started at http://127.5.83.126:43831/ using document root <none> and password file <none>
I20260923 20:15:30.614347 7864 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:30.614492 7864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:30.614815 7864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:30.617964 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "84393bf05c6a466a97b712961816a949"
format_stamp: "Formatted at 2026-09-23 20:15:30 on dist-test-slave-vl24"
server_key: "e12cb35b83a9e47c68d868be66c157c3"
server_key_iv: "22cd7ac9043d38f8c157e003a02a8f0b"
server_key_version: "encryptionkey@0"
I20260923 20:15:30.618767 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "84393bf05c6a466a97b712961816a949"
format_stamp: "Formatted at 2026-09-23 20:15:30 on dist-test-slave-vl24"
server_key: "e12cb35b83a9e47c68d868be66c157c3"
server_key_iv: "22cd7ac9043d38f8c157e003a02a8f0b"
server_key_version: "encryptionkey@0"
I20260923 20:15:30.623966 7864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:15:30.627812 7879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:30.629343 7864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:30.629535 7864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "84393bf05c6a466a97b712961816a949"
format_stamp: "Formatted at 2026-09-23 20:15:30 on dist-test-slave-vl24"
server_key: "e12cb35b83a9e47c68d868be66c157c3"
server_key_iv: "22cd7ac9043d38f8c157e003a02a8f0b"
server_key_version: "encryptionkey@0"
I20260923 20:15:30.629760 7864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:30.652094 7864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:30.656644 7864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:30.656962 7864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:30.675794 7864 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:36611
I20260923 20:15:30.675822 7930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:36611 every 8 connection(s)
I20260923 20:15:30.677336 7864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:30.681138 7931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:30.685489 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 7864
I20260923 20:15:30.685699 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:30.686530 5453 external_mini_cluster.cc:1428] Setting key cb069971a983ce5642f242944ceb7de9
I20260923 20:15:30.692850 7931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949: Bootstrap starting.
I20260923 20:15:30.696760 7931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:30.697953 7931 log.cc:824] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:30.700764 7931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949: No bootstrap required, opened a new log
Sep 23 20:15:30 dist-test-slave-vl24 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:30.708509 7931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84393bf05c6a466a97b712961816a949" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36611 } }
I20260923 20:15:30.708967 7931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:30.709105 7931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84393bf05c6a466a97b712961816a949, State: Initialized, Role: FOLLOWER
I20260923 20:15:30.709717 7931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84393bf05c6a466a97b712961816a949" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36611 } }
I20260923 20:15:30.709961 7931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:30.710011 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:30.688644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.5.83.126:36611)
0923 20:15:30.689462 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:30.689474 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:30.689518 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:15:30.691283 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:30.691309 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:30.691695 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:30.692016 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:30.694117 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:30.695536 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:30.697378 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:30.698063 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:30.698289 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:30.702406 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:30.702414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:30.702432 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:30.702470 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:30.705654 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:30.706338 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:30.706344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:30.706354 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:30.706520 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:30.706928 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:30.706931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:30.706934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:30.707457 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:30.707793 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:30.708238 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 20:15:30.708703 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":388,"threads_started":1}
I20260923 20:15:30.710110 7931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:30.710268 7931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:30.712695 7931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84393bf05c6a466a97b712961816a949" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36611 } }
I20260923 20:15:30.713181 7931 leader_election.cc:302] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84393bf05c6a466a97b712961816a949; no voters:
I20260923 20:15:30.713717 7931 leader_election.cc:288] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:30.713845 7936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:30.714581 7936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [term 1 LEADER]: Becoming Leader. State: Replica: 84393bf05c6a466a97b712961816a949, State: Running, Role: LEADER
I20260923 20:15:30.715093 7936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84393bf05c6a466a97b712961816a949" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36611 } }
I20260923 20:15:30.715791 7931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:30.717861 7937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84393bf05c6a466a97b712961816a949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84393bf05c6a466a97b712961816a949" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36611 } } }
I20260923 20:15:30.717995 7938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84393bf05c6a466a97b712961816a949. Latest consensus state: current_term: 1 leader_uuid: "84393bf05c6a466a97b712961816a949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84393bf05c6a466a97b712961816a949" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36611 } } }
I20260923 20:15:30.718364 7937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:30.718400 7938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:30.720057 7941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:30.726884 7941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:30.735741 7941 catalog_manager.cc:1347] Generated new cluster ID: 2a23f80ae51e454298a9d2158bf71019
I20260923 20:15:30.735867 7941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:30.784390 7941 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:30.786362 7941 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:30.797637 7941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84393bf05c6a466a97b712961816a949: Generated new TSK 0
I20260923 20:15:30.798679 7941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:15:30.855657 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36611
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:31.186750 7957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:31.187042 7957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:31.187139 7957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:31.194537 7957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:31.194641 7957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:31.194769 7957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:15:31.204487 7957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:31.205835 7957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:31.207103 7957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:31.221004 7965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:31.221129 7963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:31.221031 7962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:31.222494 7957 server_base.cc:1034] running on GCE node
I20260923 20:15:31.223201 7957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:31.224150 7957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:31.225361 7957 hybrid_clock.cc:648] HybridClock initialized: now 1790194531225320 us; error 30 us; skew 500 ppm
I20260923 20:15:31.225685 7957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:31 dist-test-slave-vl24 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194531, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:31.228901 7957 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:15:31.230265 7957 webserver.cc:460] Webserver started at http://127.5.83.65:33327/ using document root <none> and password file <none>
I20260923 20:15:31.230929 7957 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:31.231072 7957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:31.231382 7957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:31.234447 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "b05e21e3d58e490c9b096e098c634180"
format_stamp: "Formatted at 2026-09-23 20:15:31 on dist-test-slave-vl24"
server_key: "997c9acba14a334e93076d55bef7556c"
server_key_iv: "c427e120cf840250bee746b28760ffaa"
server_key_version: "encryptionkey@0"
I20260923 20:15:31.235234 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "b05e21e3d58e490c9b096e098c634180"
format_stamp: "Formatted at 2026-09-23 20:15:31 on dist-test-slave-vl24"
server_key: "997c9acba14a334e93076d55bef7556c"
server_key_iv: "c427e120cf840250bee746b28760ffaa"
server_key_version: "encryptionkey@0"
I20260923 20:15:31.240214 7957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:15:31.243654 7972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:31.244864 7957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:15:31.245023 7957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "b05e21e3d58e490c9b096e098c634180"
format_stamp: "Formatted at 2026-09-23 20:15:31 on dist-test-slave-vl24"
server_key: "997c9acba14a334e93076d55bef7556c"
server_key_iv: "c427e120cf840250bee746b28760ffaa"
server_key_version: "encryptionkey@0"
I20260923 20:15:31.245220 7957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:31.264492 7957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:31.268918 7957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:31.269233 7957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:31.270196 7957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:31.271512 7957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:31.271615 7957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:31.271771 7957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:31.271843 7957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:31.304467 7957 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:45775
I20260923 20:15:31.304617 8084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:45775 every 8 connection(s)
I20260923 20:15:31.305769 7957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:15:31.314986 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 7957
I20260923 20:15:31.315178 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:15:31.316015 5453 external_mini_cluster.cc:1428] Setting key b356b0e18b601964b92d477f94dd7f46
Sep 23 20:15:31 dist-test-slave-vl24 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194531, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:31.327625 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:31.308331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:38769 (local address 127.5.83.126:36611)
0923 20:15:31.308772 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:31.308776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:31.309730 (+ 954us) server_negotiation.cc:407] Connection header received
0923 20:15:31.311522 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:31.311527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:31.311623 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:31.311740 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:31.313229 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:31.313956 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:31.316214 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:31.316813 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:31.316955 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:31.320637 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:31.320646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:31.320653 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:31.320699 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:31.324235 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:31.325275 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:31.325279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:31.325281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:31.325396 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:31.325885 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:31.325888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:31.325890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:31.326151 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:31.326257 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:31.327196 (+ 939us) server_negotiation.cc:299] Negotiation successful
0923 20:15:31.327361 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":240,"threads_started":1}
I20260923 20:15:31.329252 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:31.308913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36611 (local address 127.5.83.65:38769)
0923 20:15:31.309575 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:31.309621 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:31.311258 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:31.311963 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:31.311971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:31.312453 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:31.313070 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:31.313087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:31.314114 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:31.314119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:31.316002 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:31.316018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:31.317139 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:31.317144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:31.317334 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:31.318088 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:31.318116 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:31.320412 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:31.324391 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:31.324402 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:31.324425 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:31.325143 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:31.325505 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:31.325512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:31.325519 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:31.325771 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:31.326271 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:31.326282 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:31.326764 (+ 482us) client_negotiation.cc:769] Sending connection context
0923 20:15:31.327084 (+ 320us) client_negotiation.cc:240] Negotiation successful
0923 20:15:31.327710 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":260,"threads_started":1}
I20260923 20:15:31.330910 8085 heartbeater.cc:347] Connected to a master server at 127.5.83.126:36611
I20260923 20:15:31.331231 8085 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:31.331967 8085 heartbeater.cc:511] Master 127.5.83.126:36611 requested a full tablet report, sending...
I20260923 20:15:31.334266 7896 ts_manager.cc:194] Registered new tserver with Master: b05e21e3d58e490c9b096e098c634180 (127.5.83.65:45775)
I20260923 20:15:31.335706 7896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:38769
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:15:31.361881 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36611
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:31.696213 8091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:31.696493 8091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:31.696661 8091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:31.704066 8091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:31.704172 8091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:31.704283 8091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:15:31.714071 8091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:31.715283 8091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:31.716600 8091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:31.729537 8097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:31.729497 8099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:31.729626 8096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:31.730191 8091 server_base.cc:1034] running on GCE node
I20260923 20:15:31.730760 8091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:31.731678 8091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:31.732896 8091 hybrid_clock.cc:648] HybridClock initialized: now 1790194531732846 us; error 42 us; skew 500 ppm
I20260923 20:15:31.733234 8091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:31 dist-test-slave-vl24 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194531, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:31.736392 8091 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:15:31.737883 8091 webserver.cc:460] Webserver started at http://127.5.83.66:41539/ using document root <none> and password file <none>
I20260923 20:15:31.738587 8091 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:31.738744 8091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:31.739066 8091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:31.742115 8091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "828fa78a16fe4aca8017a9d07bd33784"
format_stamp: "Formatted at 2026-09-23 20:15:31 on dist-test-slave-vl24"
server_key: "d582b2e68e5762ba3d20636d4075d3d1"
server_key_iv: "5ca1e685d39e26a404917d358c79288f"
server_key_version: "encryptionkey@0"
I20260923 20:15:31.742910 8091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "828fa78a16fe4aca8017a9d07bd33784"
format_stamp: "Formatted at 2026-09-23 20:15:31 on dist-test-slave-vl24"
server_key: "d582b2e68e5762ba3d20636d4075d3d1"
server_key_iv: "5ca1e685d39e26a404917d358c79288f"
server_key_version: "encryptionkey@0"
I20260923 20:15:31.748173 8091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:15:31.751819 8106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:31.753129 8091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:31.753305 8091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "828fa78a16fe4aca8017a9d07bd33784"
format_stamp: "Formatted at 2026-09-23 20:15:31 on dist-test-slave-vl24"
server_key: "d582b2e68e5762ba3d20636d4075d3d1"
server_key_iv: "5ca1e685d39e26a404917d358c79288f"
server_key_version: "encryptionkey@0"
I20260923 20:15:31.753515 8091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:31.767974 8091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:31.772487 8091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:31.772837 8091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:31.773799 8091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:31.775137 8091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:31.775241 8091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:31.775393 8091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:31.775471 8091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:31.808503 8091 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:45133
I20260923 20:15:31.808590 8218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:45133 every 8 connection(s)
I20260923 20:15:31.809846 8091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:15:31.814185 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 8091
I20260923 20:15:31.814363 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:15:31.814814 5453 external_mini_cluster.cc:1428] Setting key ffa898cca47d4890170a49476a5ff9fb
Sep 23 20:15:31 dist-test-slave-vl24 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194531, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:31.831676 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:31.812292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:58425 (local address 127.5.83.126:36611)
0923 20:15:31.812458 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:31.812462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:31.813574 (+ 1112us) server_negotiation.cc:407] Connection header received
0923 20:15:31.815620 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:31.815626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:31.815715 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:31.815823 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:31.817528 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:31.818230 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:31.819723 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:31.820159 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:31.820292 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:31.824245 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:31.824255 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:31.824263 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:31.824308 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:31.828206 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:31.829037 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:31.829047 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:31.829054 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:31.829239 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:31.829682 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:31.829690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:31.829696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:31.830141 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:31.830335 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:31.831209 (+ 874us) server_negotiation.cc:299] Negotiation successful
0923 20:15:31.831400 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 20:15:31.833226 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:31.812738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36611 (local address 127.5.83.66:58425)
0923 20:15:31.813423 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:31.813459 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:31.815260 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:31.816086 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:31.816119 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:31.816790 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:31.817363 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:31.817380 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:31.818388 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:31.818392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:31.819575 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:31.819585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:31.820374 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:31.820380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:31.820682 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:31.821388 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:31.821409 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:31.824020 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:31.828366 (+ 4346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:31.828374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:31.828387 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:31.828886 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:31.829368 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:31.829371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:31.829373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:31.829552 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:31.830295 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:31.830306 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:31.830705 (+ 399us) client_negotiation.cc:769] Sending connection context
0923 20:15:31.830978 (+ 273us) client_negotiation.cc:240] Negotiation successful
0923 20:15:31.831618 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":261,"threads_started":1}
I20260923 20:15:31.834823 8219 heartbeater.cc:347] Connected to a master server at 127.5.83.126:36611
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
I20260923 20:15:31.835160 8219 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:31.835868 8219 heartbeater.cc:511] Master 127.5.83.126:36611 requested a full tablet report, sending...
I20260923 20:15:31.837472 7896 ts_manager.cc:194] Registered new tserver with Master: 828fa78a16fe4aca8017a9d07bd33784 (127.5.83.66:45133)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
I20260923 20:15:31.838523 7896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:58425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:15:31.859951 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36611
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:32861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:32.184284 8224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:32.184646 8224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:32.184824 8224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:32.192260 8224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:32.192360 8224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:32.192624 8224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:15:32.202641 8224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:32861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:32.203917 8224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:32.205219 8224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:32.218122 8229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:32.218084 8230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:32.218271 8232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:32.219236 8224 server_base.cc:1034] running on GCE node
I20260923 20:15:32.219792 8224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:32.220712 8224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:32.221927 8224 hybrid_clock.cc:648] HybridClock initialized: now 1790194532221889 us; error 49 us; skew 500 ppm
I20260923 20:15:32.222268 8224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:32 dist-test-slave-vl24 krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194532, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:32.225296 8224 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:15:32.226625 8224 webserver.cc:460] Webserver started at http://127.5.83.67:37347/ using document root <none> and password file <none>
I20260923 20:15:32.227263 8224 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:32.227401 8224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:32.227696 8224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:32.230528 8224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "2b2bb3217294402f85fd7e20d4e22897"
format_stamp: "Formatted at 2026-09-23 20:15:32 on dist-test-slave-vl24"
server_key: "10a4d9c9b3fc51712a591cba624c5098"
server_key_iv: "6200c0d71f236ce37ca188609a2cb368"
server_key_version: "encryptionkey@0"
I20260923 20:15:32.231280 8224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "2b2bb3217294402f85fd7e20d4e22897"
format_stamp: "Formatted at 2026-09-23 20:15:32 on dist-test-slave-vl24"
server_key: "10a4d9c9b3fc51712a591cba624c5098"
server_key_iv: "6200c0d71f236ce37ca188609a2cb368"
server_key_version: "encryptionkey@0"
I20260923 20:15:32.236336 8224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 20:15:32.239766 8239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:32.241039 8224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:32.241206 8224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "2b2bb3217294402f85fd7e20d4e22897"
format_stamp: "Formatted at 2026-09-23 20:15:32 on dist-test-slave-vl24"
server_key: "10a4d9c9b3fc51712a591cba624c5098"
server_key_iv: "6200c0d71f236ce37ca188609a2cb368"
server_key_version: "encryptionkey@0"
I20260923 20:15:32.241397 8224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:32.270139 8224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:32.274082 8224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:32.274364 8224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:32.275265 8224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:32.276512 8224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:32.276630 8224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:32.276810 8224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:32.276888 8224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:32.310385 8224 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:36969
I20260923 20:15:32.310420 8351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:36969 every 8 connection(s)
I20260923 20:15:32.311620 8224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:15:32.318637 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 8224
I20260923 20:15:32.318806 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:15:32.319240 5453 external_mini_cluster.cc:1428] Setting key 3a8ef3e399d67b5b0073369048667ab2
Sep 23 20:15:32 dist-test-slave-vl24 krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194532, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:32.332443 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:32.314172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:54839 (local address 127.5.83.126:36611)
0923 20:15:32.314351 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:32.314357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:32.315745 (+ 1388us) server_negotiation.cc:407] Connection header received
0923 20:15:32.317865 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:32.317871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:32.317970 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:32.318081 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:32.319735 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:32.320752 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:32.322656 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:32.323061 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:32.323198 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:32.326730 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:32.326735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:32.326739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:32.326772 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:32.329461 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:32.330174 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:32.330181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:32.330188 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:32.330323 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:32.330759 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:32.330766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:32.330769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:32.331030 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:32.331190 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:32.331868 (+ 678us) server_negotiation.cc:299] Negotiation successful
0923 20:15:32.332085 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:15:32.333781 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:32.314852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36611 (local address 127.5.83.67:54839)
0923 20:15:32.315580 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:32.315619 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:32.317589 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:32.318322 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:32.318330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:32.318946 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:32.319560 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:32.319579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:32.320918 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:32.320922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:32.322504 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:32.322515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:32.323298 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:32.323305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:32.323545 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:32.324222 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:32.324245 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:32.326502 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:32.329612 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:32.329621 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:32.329638 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:32.330052 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:32.330443 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:32.330448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:32.330452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:32.330646 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:32.331164 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:32.331180 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:32.331515 (+ 335us) client_negotiation.cc:769] Sending connection context
0923 20:15:32.331780 (+ 265us) client_negotiation.cc:240] Negotiation successful
0923 20:15:32.332325 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":315,"threads_started":1}
I20260923 20:15:32.335263 8352 heartbeater.cc:347] Connected to a master server at 127.5.83.126:36611
I20260923 20:15:32.335563 8352 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:32.336262 8352 heartbeater.cc:511] Master 127.5.83.126:36611 requested a full tablet report, sending...
I20260923 20:15:32.337630 7896 ts_manager.cc:194] Registered new tserver with Master: 2b2bb3217294402f85fd7e20d4e22897 (127.5.83.67:36969)
I20260923 20:15:32.338438 7896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:54839
I20260923 20:15:32.338728 8085 heartbeater.cc:503] Master 127.5.83.126:36611 was elected leader, sending a full tablet report...
I20260923 20:15:32.347080 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 20:15:32.379434 8087 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:15:32.376959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.5.83.126:36611)
0923 20:15:32.377189 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:32.377198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:32.377592 (+ 394us) server_negotiation.cc:407] Connection header received
0923 20:15:32.377745 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:32.377748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:32.377815 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:32.377902 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:32.379136 (+ 1234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60624: BlockingRecv error: recv got EOF from 127.0.0.1:60624 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:15:32.382309 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 7957
W20260923 20:15:32.385058 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 7957: Not authorized: Client connection negotiation failed: client connection to 127.5.83.65:45775: server requires authentication, but client does not have Kerberos credentials available
I20260923 20:15:32.385268 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 7957
I20260923 20:15:32.401585 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 8091
W20260923 20:15:32.404135 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 8091: Not authorized: Client connection negotiation failed: client connection to 127.5.83.66:45133: server requires authentication, but client does not have Kerberos credentials available
I20260923 20:15:32.404281 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 8091
I20260923 20:15:32.419943 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 8224
W20260923 20:15:32.422667 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 8224: Not authorized: Client connection negotiation failed: client connection to 127.5.83.67:36969: server requires authentication, but client does not have Kerberos credentials available
I20260923 20:15:32.422832 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 8224
I20260923 20:15:32.438583 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 7864
I20260923 20:15:32.440006 7926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:32.541843 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 7864
2026-09-23T20:15:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3587 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:15:32 dist-test-slave-vl24 krb5kdc[8369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:15:32 dist-test-slave-vl24 krb5kdc[8369](info): set up 2 sockets
Sep 23 20:15:32 dist-test-slave-vl24 krb5kdc[8369](info): commencing operation
krb5kdc: starting...
W20260923 20:15:33.646059 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:15:33 dist-test-slave-vl24 krb5kdc[8369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:15:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:15:33.839433 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:33169
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:33169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:34.160014 8382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:34.160382 8382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:34.160597 8382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:34.167824 8382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:34.167971 8382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:34.168083 8382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:34.168200 8382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:34.168291 8382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:15:34.178203 8382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:33169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:33169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:34.179589 8382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:34.181006 8382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:34.192770 8387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:34.192868 8390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:34.192902 8388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:34.193674 8382 server_base.cc:1034] running on GCE node
I20260923 20:15:34.194352 8382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:34.195667 8382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:34.196887 8382 hybrid_clock.cc:648] HybridClock initialized: now 1790194534196853 us; error 53 us; skew 500 ppm
I20260923 20:15:34.197237 8382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:34 dist-test-slave-vl24 krb5kdc[8369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194534, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:34.200161 8382 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:15:34.201601 8382 webserver.cc:460] Webserver started at http://127.5.83.126:42407/ using document root <none> and password file <none>
I20260923 20:15:34.202262 8382 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:34.202360 8382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:34.202611 8382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:34.205557 8382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "70e8c7f1048841aea1ea06cb3abea4fa"
format_stamp: "Formatted at 2026-09-23 20:15:34 on dist-test-slave-vl24"
server_key: "c5d20aa3277ec3ecfa4013130ed1368d"
server_key_iv: "3816d8e87e652231ab5ac0552667a07f"
server_key_version: "encryptionkey@0"
I20260923 20:15:34.206303 8382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "70e8c7f1048841aea1ea06cb3abea4fa"
format_stamp: "Formatted at 2026-09-23 20:15:34 on dist-test-slave-vl24"
server_key: "c5d20aa3277ec3ecfa4013130ed1368d"
server_key_iv: "3816d8e87e652231ab5ac0552667a07f"
server_key_version: "encryptionkey@0"
I20260923 20:15:34.211632 8382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:15:34.215296 8397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:34.216780 8382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:34.216961 8382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "70e8c7f1048841aea1ea06cb3abea4fa"
format_stamp: "Formatted at 2026-09-23 20:15:34 on dist-test-slave-vl24"
server_key: "c5d20aa3277ec3ecfa4013130ed1368d"
server_key_iv: "3816d8e87e652231ab5ac0552667a07f"
server_key_version: "encryptionkey@0"
I20260923 20:15:34.217168 8382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:34.234877 8382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:34.238951 8382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:34.239225 8382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:34.257046 8382 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:33169
I20260923 20:15:34.257089 8448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:33169 every 8 connection(s)
I20260923 20:15:34.258395 8382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:34.262036 8449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:34.268002 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 8382
I20260923 20:15:34.268241 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:34.269059 5453 external_mini_cluster.cc:1428] Setting key eff820890d54e9c6d06a393924fb1ca7
I20260923 20:15:34.273383 8449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa: Bootstrap starting.
I20260923 20:15:34.277112 8449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:34.278324 8449 log.cc:824] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:34.281075 8449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa: No bootstrap required, opened a new log
Sep 23 20:15:34 dist-test-slave-vl24 krb5kdc[8369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:34.288913 8449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e8c7f1048841aea1ea06cb3abea4fa" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 33169 } }
I20260923 20:15:34.289312 8449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:34.289433 8449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e8c7f1048841aea1ea06cb3abea4fa, State: Initialized, Role: FOLLOWER
I20260923 20:15:34.290127 8449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e8c7f1048841aea1ea06cb3abea4fa" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 33169 } }
I20260923 20:15:34.290397 8449 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:34.290534 8449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:34.290704 8449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:34.291618 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:34.270916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49012 (local address 127.5.83.126:33169)
0923 20:15:34.271682 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:34.271694 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:34.271733 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:15:34.273492 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:34.273523 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:34.273900 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:34.274209 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:34.276038 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:34.277456 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:34.279242 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:34.279912 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:34.280096 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:34.284048 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:34.284055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:34.284068 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:34.284102 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:34.287164 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:34.287865 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:34.287879 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:34.287885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:34.288045 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:34.288431 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:34.288438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:34.288445 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:34.289040 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:34.289345 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:34.289793 (+ 448us) server_negotiation.cc:299] Negotiation successful
0923 20:15:34.290218 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":361,"threads_started":1}
I20260923 20:15:34.293787 8449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e8c7f1048841aea1ea06cb3abea4fa" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 33169 } }
I20260923 20:15:34.294356 8449 leader_election.cc:302] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e8c7f1048841aea1ea06cb3abea4fa; no voters:
I20260923 20:15:34.294968 8449 leader_election.cc:288] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:34.295109 8454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:34.295836 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:15:34.296806 8454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [term 1 LEADER]: Becoming Leader. State: Replica: 70e8c7f1048841aea1ea06cb3abea4fa, State: Running, Role: LEADER
I20260923 20:15:34.297266 8449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:34.297436 8454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e8c7f1048841aea1ea06cb3abea4fa" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 33169 } }
I20260923 20:15:34.300668 8456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e8c7f1048841aea1ea06cb3abea4fa. Latest consensus state: current_term: 1 leader_uuid: "70e8c7f1048841aea1ea06cb3abea4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e8c7f1048841aea1ea06cb3abea4fa" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 33169 } } }
I20260923 20:15:34.301105 8456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:34.300736 8455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70e8c7f1048841aea1ea06cb3abea4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e8c7f1048841aea1ea06cb3abea4fa" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 33169 } } }
I20260923 20:15:34.301770 8455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:34.310371 8462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:34.316422 8462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:34.335227 8462 catalog_manager.cc:1347] Generated new cluster ID: abcd883f4c3b4efea82d2cf9a36b7886
I20260923 20:15:34.335420 8462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:34.351661 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:34.342180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.5.83.126:33169)
0923 20:15:34.342410 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:34.342416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:34.343013 (+ 597us) server_negotiation.cc:407] Connection header received
0923 20:15:34.343190 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:34.343195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:34.343270 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:34.343377 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:34.344697 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:34.345785 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:34.347752 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:34.348490 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:34.348660 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:34.349783 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:34.349790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:34.349795 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:34.350809 (+ 1014us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:34.350891 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:34.350925 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:34.351048 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:34.351178 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 20:15:34.351288 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":59}
I20260923 20:15:34.371536 8462 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:34.373329 8462 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:34.379518 8462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70e8c7f1048841aea1ea06cb3abea4fa: Generated new TSK 0
I20260923 20:15:34.380555 8462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:15:34.394235 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 8382
I20260923 20:15:34.395462 8444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:34.488441 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 8382
2026-09-23T20:15:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1941 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:15:34 dist-test-slave-vl24 krb5kdc[8483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:15:34 dist-test-slave-vl24 krb5kdc[8483](info): set up 2 sockets
Sep 23 20:15:34 dist-test-slave-vl24 krb5kdc[8483](info): commencing operation
krb5kdc: starting...
W20260923 20:15:35.570236 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:15:35 dist-test-slave-vl24 krb5kdc[8483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:15:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:15:35.779894 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:37393
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:37393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:36.098805 8496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:36.099092 8496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:36.099300 8496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:36.106665 8496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:36.106796 8496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:36.106895 8496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:36.107005 8496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:36.107084 8496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:15:36.117187 8496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:37393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:37393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:36.118587 8496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:36.119917 8496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:36.131801 8501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:36.131748 8504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:36.131770 8502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:36.133069 8496 server_base.cc:1034] running on GCE node
I20260923 20:15:36.133882 8496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:36.135344 8496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:36.136619 8496 hybrid_clock.cc:648] HybridClock initialized: now 1790194536136575 us; error 40 us; skew 500 ppm
I20260923 20:15:36.136996 8496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:36 dist-test-slave-vl24 krb5kdc[8483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194536, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:36.149684 8496 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:15:36.151448 8496 webserver.cc:460] Webserver started at http://127.5.83.126:36353/ using document root <none> and password file <none>
I20260923 20:15:36.152200 8496 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:36.152359 8496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:36.152766 8496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:36.155910 8496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "eb717e87aeed482ba7f1edf7258287cf"
format_stamp: "Formatted at 2026-09-23 20:15:36 on dist-test-slave-vl24"
server_key: "1d858d3668ab9ae3e88da51ec3324146"
server_key_iv: "b92c05ab30c3a95c364264aee5cb586b"
server_key_version: "encryptionkey@0"
I20260923 20:15:36.156797 8496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "eb717e87aeed482ba7f1edf7258287cf"
format_stamp: "Formatted at 2026-09-23 20:15:36 on dist-test-slave-vl24"
server_key: "1d858d3668ab9ae3e88da51ec3324146"
server_key_iv: "b92c05ab30c3a95c364264aee5cb586b"
server_key_version: "encryptionkey@0"
I20260923 20:15:36.162158 8496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:15:36.165913 8511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:36.167543 8496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:15:36.167769 8496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "eb717e87aeed482ba7f1edf7258287cf"
format_stamp: "Formatted at 2026-09-23 20:15:36 on dist-test-slave-vl24"
server_key: "1d858d3668ab9ae3e88da51ec3324146"
server_key_iv: "b92c05ab30c3a95c364264aee5cb586b"
server_key_version: "encryptionkey@0"
I20260923 20:15:36.167979 8496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:36.191206 8496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:36.195398 8496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:36.195698 8496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:36.214569 8496 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:37393
I20260923 20:15:36.214598 8562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:37393 every 8 connection(s)
I20260923 20:15:36.216079 8496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:36.220268 8563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:36.220366 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 8496
I20260923 20:15:36.220654 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:36.221279 5453 external_mini_cluster.cc:1428] Setting key 37afa71c4281b0c9c2a78f34e9186b6c
I20260923 20:15:36.234261 8563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf: Bootstrap starting.
Sep 23 20:15:36 dist-test-slave-vl24 krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:36.238291 8563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:36.239609 8563 log.cc:824] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:36.242486 8563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf: No bootstrap required, opened a new log
I20260923 20:15:36.245689 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:36.223397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51586 (local address 127.5.83.126:37393)
0923 20:15:36.224195 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:36.224207 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:36.224248 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:15:36.226208 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:36.226231 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:36.226627 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:36.226968 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:36.228688 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:36.230674 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:36.232489 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:36.233601 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:36.233817 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:36.238515 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:36.238523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:36.238555 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:36.238588 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:36.241716 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:36.242414 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:36.242420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:36.242427 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:36.242606 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:36.242981 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:36.242990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:36.242994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:36.243495 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:36.243814 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:36.244286 (+ 472us) server_negotiation.cc:299] Negotiation successful
0923 20:15:36.244721 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":369,"threads_started":1}
I20260923 20:15:36.250236 8563 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb717e87aeed482ba7f1edf7258287cf" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37393 } }
I20260923 20:15:36.250615 8563 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:36.250720 8563 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb717e87aeed482ba7f1edf7258287cf, State: Initialized, Role: FOLLOWER
I20260923 20:15:36.251256 8563 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb717e87aeed482ba7f1edf7258287cf" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37393 } }
I20260923 20:15:36.251497 8563 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:36.251650 8563 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:36.251771 8563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:36.254276 8563 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb717e87aeed482ba7f1edf7258287cf" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37393 } }
I20260923 20:15:36.254712 8563 leader_election.cc:302] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb717e87aeed482ba7f1edf7258287cf; no voters:
I20260923 20:15:36.255236 8563 leader_election.cc:288] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:36.255383 8568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:36.256049 8568 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [term 1 LEADER]: Becoming Leader. State: Replica: eb717e87aeed482ba7f1edf7258287cf, State: Running, Role: LEADER
I20260923 20:15:36.256649 8568 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb717e87aeed482ba7f1edf7258287cf" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37393 } }
I20260923 20:15:36.257562 8563 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:36.259258 8569 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb717e87aeed482ba7f1edf7258287cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb717e87aeed482ba7f1edf7258287cf" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37393 } } }
I20260923 20:15:36.259428 8570 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb717e87aeed482ba7f1edf7258287cf. Latest consensus state: current_term: 1 leader_uuid: "eb717e87aeed482ba7f1edf7258287cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb717e87aeed482ba7f1edf7258287cf" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37393 } } }
I20260923 20:15:36.259716 8569 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:36.259774 8570 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:36.261808 8573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:36.267594 8573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:36.276119 8573 catalog_manager.cc:1347] Generated new cluster ID: 1794e647c36045bdaf930cc806dd1784
I20260923 20:15:36.276227 8573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:36.287196 8573 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:36.288584 8573 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:36.295733 8573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb717e87aeed482ba7f1edf7258287cf: Generated new TSK 0
I20260923 20:15:36.296665 8573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:15:36.352752 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37393
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:36.684501 8589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:36.684895 8589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:36.685072 8589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:36.692435 8589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:36.692607 8589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:36.692785 8589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:15:36.702682 8589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:36.704182 8589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:36.705788 8589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:36.719266 8595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:36.719292 8594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:36.719760 8597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:36.720209 8589 server_base.cc:1034] running on GCE node
I20260923 20:15:36.720854 8589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:36.721764 8589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:36.722967 8589 hybrid_clock.cc:648] HybridClock initialized: now 1790194536722925 us; error 41 us; skew 500 ppm
I20260923 20:15:36.723268 8589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:36 dist-test-slave-vl24 krb5kdc[8483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194536, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:36.726399 8589 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:15:36.727890 8589 webserver.cc:460] Webserver started at http://127.5.83.65:33891/ using document root <none> and password file <none>
I20260923 20:15:36.728670 8589 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:36.728813 8589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:36.729140 8589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:36.732012 8589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "4519299a0bd64e18a233ea78d4671d64"
format_stamp: "Formatted at 2026-09-23 20:15:36 on dist-test-slave-vl24"
server_key: "fd87978c5d2f340f0c1587a4540ad139"
server_key_iv: "8b0295c0d442bc836a4b4f641f46095a"
server_key_version: "encryptionkey@0"
I20260923 20:15:36.732877 8589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "4519299a0bd64e18a233ea78d4671d64"
format_stamp: "Formatted at 2026-09-23 20:15:36 on dist-test-slave-vl24"
server_key: "fd87978c5d2f340f0c1587a4540ad139"
server_key_iv: "8b0295c0d442bc836a4b4f641f46095a"
server_key_version: "encryptionkey@0"
I20260923 20:15:36.738309 8589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:15:36.742002 8604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:36.743366 8589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:15:36.743546 8589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "4519299a0bd64e18a233ea78d4671d64"
format_stamp: "Formatted at 2026-09-23 20:15:36 on dist-test-slave-vl24"
server_key: "fd87978c5d2f340f0c1587a4540ad139"
server_key_iv: "8b0295c0d442bc836a4b4f641f46095a"
server_key_version: "encryptionkey@0"
I20260923 20:15:36.743747 8589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:36.766589 8589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:36.770857 8589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:36.771196 8589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:36.772192 8589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:36.773643 8589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:36.773743 8589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:36.773895 8589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:36.773967 8589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:36.807801 8589 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:43139
I20260923 20:15:36.807893 8716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:43139 every 8 connection(s)
I20260923 20:15:36.809203 8589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:15:36.814139 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 8589
I20260923 20:15:36.814366 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:15:36.815205 5453 external_mini_cluster.cc:1428] Setting key d7adbda677051e25263fad8e7e20fb13
Sep 23 20:15:36 dist-test-slave-vl24 krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194536, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:36.829998 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:36.811657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:59835 (local address 127.5.83.126:37393)
0923 20:15:36.812092 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:36.812096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:36.813088 (+ 992us) server_negotiation.cc:407] Connection header received
0923 20:15:36.815673 (+ 2585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:36.815678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:36.815748 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:36.815882 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:36.817661 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:36.818374 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:36.819947 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:36.820344 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:36.820490 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:36.824176 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:36.824181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:36.824184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:36.824216 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:36.826870 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:36.827651 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:36.827655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:36.827658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:36.827770 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:36.828203 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:36.828207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:36.828210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:36.828467 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:36.828651 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:36.829393 (+ 742us) server_negotiation.cc:299] Negotiation successful
0923 20:15:36.829628 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":206,"threads_started":1}
I20260923 20:15:36.830978 8720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:36.812183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:37393 (local address 127.5.83.65:59835)
0923 20:15:36.812918 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:36.812960 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:36.815364 (+ 2404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:36.816134 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:36.816141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:36.816892 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:36.817486 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:36.817503 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:36.818528 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:36.818535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:36.819761 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:36.819772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:36.820609 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:36.820619 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:36.820855 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:36.821549 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:36.821571 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:36.823942 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:36.827019 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:36.827028 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:36.827079 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:36.827512 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:36.827884 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:36.827889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:36.827893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:36.828085 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:36.828654 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:36.828665 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:36.829008 (+ 343us) client_negotiation.cc:769] Sending connection context
0923 20:15:36.829286 (+ 278us) client_negotiation.cc:240] Negotiation successful
0923 20:15:36.829827 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":324,"threads_started":1}
I20260923 20:15:36.832762 8717 heartbeater.cc:347] Connected to a master server at 127.5.83.126:37393
I20260923 20:15:36.833145 8717 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
I20260923 20:15:36.834028 8717 heartbeater.cc:511] Master 127.5.83.126:37393 requested a full tablet report, sending...
I20260923 20:15:36.836102 8528 ts_manager.cc:194] Registered new tserver with Master: 4519299a0bd64e18a233ea78d4671d64 (127.5.83.65:43139)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
I20260923 20:15:36.837656 8528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:59835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:15:36.861600 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37393
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:37.194511 8723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:37.194846 8723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:37.195024 8723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:37.202450 8723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:37.202610 8723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:37.202801 8723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:15:37.212721 8723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:37.214182 8723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:37.215499 8723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:37.228825 8729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:37.229228 8731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:37.228905 8728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:37.230252 8723 server_base.cc:1034] running on GCE node
I20260923 20:15:37.230964 8723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:37.231927 8723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:37.233119 8723 hybrid_clock.cc:648] HybridClock initialized: now 1790194537233078 us; error 32 us; skew 500 ppm
I20260923 20:15:37.233438 8723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:37 dist-test-slave-vl24 krb5kdc[8483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194537, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:37.236698 8723 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:15:37.238165 8723 webserver.cc:460] Webserver started at http://127.5.83.66:44469/ using document root <none> and password file <none>
I20260923 20:15:37.238884 8723 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:37.239025 8723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:37.239333 8723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:37.242549 8723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "2b5657ccc69f4ef898ac81b176e430d1"
format_stamp: "Formatted at 2026-09-23 20:15:37 on dist-test-slave-vl24"
server_key: "66ba63e4d34216c2bf5d9c6a74cb3f87"
server_key_iv: "dbdb80db9173164467dfafa2decd4407"
server_key_version: "encryptionkey@0"
I20260923 20:15:37.243338 8723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "2b5657ccc69f4ef898ac81b176e430d1"
format_stamp: "Formatted at 2026-09-23 20:15:37 on dist-test-slave-vl24"
server_key: "66ba63e4d34216c2bf5d9c6a74cb3f87"
server_key_iv: "dbdb80db9173164467dfafa2decd4407"
server_key_version: "encryptionkey@0"
I20260923 20:15:37.248879 8723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:15:37.252141 8738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:37.253571 8723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:37.253750 8723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "2b5657ccc69f4ef898ac81b176e430d1"
format_stamp: "Formatted at 2026-09-23 20:15:37 on dist-test-slave-vl24"
server_key: "66ba63e4d34216c2bf5d9c6a74cb3f87"
server_key_iv: "dbdb80db9173164467dfafa2decd4407"
server_key_version: "encryptionkey@0"
I20260923 20:15:37.253942 8723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:37.279700 8723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:37.284041 8723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:37.284375 8723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:37.285427 8723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:37.286926 8723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:37.287001 8723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:37.287173 8723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:37.287225 8723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:37.319180 8723 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:39041
I20260923 20:15:37.319271 8850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:39041 every 8 connection(s)
I20260923 20:15:37.320490 8723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:15:37.321545 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 8723
I20260923 20:15:37.321787 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:15:37.322227 5453 external_mini_cluster.cc:1428] Setting key 4c9049cef9683ce89577b6405ee115ad
Sep 23 20:15:37 dist-test-slave-vl24 krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194537, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
I20260923 20:15:37.342935 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:37.323352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:47053 (local address 127.5.83.126:37393)
0923 20:15:37.324356 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:37.324363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:37.324887 (+ 524us) server_negotiation.cc:407] Connection header received
0923 20:15:37.326772 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:37.326780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:37.326883 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:37.327016 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:37.328511 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.329265 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.331718 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.332228 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.332348 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:37.336926 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:37.336932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:37.336935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:37.336971 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:37.339745 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:37.340521 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:37.340562 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:37.340568 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:37.340708 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:37.341152 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:37.341158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:37.341161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:37.341442 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:37.341617 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:37.342354 (+ 737us) server_negotiation.cc:299] Negotiation successful
0923 20:15:37.342594 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844}
I20260923 20:15:37.343966 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:37.323897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:37393 (local address 127.5.83.66:47053)
0923 20:15:37.324718 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:37.324765 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:37.326472 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:37.327212 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:37.327220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:37.327764 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:37.328322 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:37.328340 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.329433 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.329438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:37.331510 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:37.331531 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.332474 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.332484 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:37.332882 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:37.333713 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:37.333750 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:37.336643 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:37.339912 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:37.339923 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:37.339948 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:37.340394 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:37.340823 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:37.340829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:37.340832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:37.341030 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:37.341578 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:37.341588 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:37.341983 (+ 395us) client_negotiation.cc:769] Sending connection context
0923 20:15:37.342270 (+ 287us) client_negotiation.cc:240] Negotiation successful
0923 20:15:37.342832 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":400,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
I20260923 20:15:37.345800 8851 heartbeater.cc:347] Connected to a master server at 127.5.83.126:37393
I20260923 20:15:37.346149 8851 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:37.347105 8851 heartbeater.cc:511] Master 127.5.83.126:37393 requested a full tablet report, sending...
I20260923 20:15:37.348798 8528 ts_manager.cc:194] Registered new tserver with Master: 2b5657ccc69f4ef898ac81b176e430d1 (127.5.83.66:39041)
I20260923 20:15:37.349599 8528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:47053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:15:37.369411 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37393
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:37.691776 8856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:37.692122 8856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:37.692296 8856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:37.699613 8856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:37.699779 8856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:37.699972 8856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:15:37.709966 8856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:37.711349 8856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:37.712745 8856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:37.725704 8864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:37.725771 8862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:37.725730 8861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:37.726859 8856 server_base.cc:1034] running on GCE node
I20260923 20:15:37.727541 8856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:37.728502 8856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:37.729779 8856 hybrid_clock.cc:648] HybridClock initialized: now 1790194537729719 us; error 53 us; skew 500 ppm
I20260923 20:15:37.730086 8856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:37 dist-test-slave-vl24 krb5kdc[8483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194537, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:37.733256 8856 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:15:37.734756 8856 webserver.cc:460] Webserver started at http://127.5.83.67:46395/ using document root <none> and password file <none>
I20260923 20:15:37.735464 8856 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:37.735610 8856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:37.735919 8856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:37.738893 8856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "357c8253f3434d21867cfcc263070471"
format_stamp: "Formatted at 2026-09-23 20:15:37 on dist-test-slave-vl24"
server_key: "2c26ba178d2ff1c40556ff0d2e58a602"
server_key_iv: "b462f34bdbbb8afb9e412eac5fa2aaf7"
server_key_version: "encryptionkey@0"
I20260923 20:15:37.739694 8856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "357c8253f3434d21867cfcc263070471"
format_stamp: "Formatted at 2026-09-23 20:15:37 on dist-test-slave-vl24"
server_key: "2c26ba178d2ff1c40556ff0d2e58a602"
server_key_iv: "b462f34bdbbb8afb9e412eac5fa2aaf7"
server_key_version: "encryptionkey@0"
I20260923 20:15:37.744990 8856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 20:15:37.748673 8871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:37.750011 8856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:15:37.750200 8856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "357c8253f3434d21867cfcc263070471"
format_stamp: "Formatted at 2026-09-23 20:15:37 on dist-test-slave-vl24"
server_key: "2c26ba178d2ff1c40556ff0d2e58a602"
server_key_iv: "b462f34bdbbb8afb9e412eac5fa2aaf7"
server_key_version: "encryptionkey@0"
I20260923 20:15:37.750416 8856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:37.764835 8856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:37.769193 8856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:37.769446 8856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:37.770337 8856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:37.771685 8856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:37.771760 8856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:37.771842 8856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:37.771889 8856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:37.805212 8856 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:41819
I20260923 20:15:37.805248 8983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:41819 every 8 connection(s)
I20260923 20:15:37.806553 8856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:15:37.816962 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 8856
I20260923 20:15:37.817175 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:15:37.817720 5453 external_mini_cluster.cc:1428] Setting key 060c903da705dbee2f7cd52704728c28
Sep 23 20:15:37 dist-test-slave-vl24 krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194537, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:37.826424 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:37.809247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:41211 (local address 127.5.83.126:37393)
0923 20:15:37.809421 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:37.809426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:37.810768 (+ 1342us) server_negotiation.cc:407] Connection header received
0923 20:15:37.812554 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:37.812560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:37.812678 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:37.812794 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:37.814226 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.814969 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.816489 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.817192 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.817321 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:37.820966 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:37.820972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:37.820977 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:37.821017 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:37.823449 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:37.824145 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:37.824153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:37.824156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:37.824289 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:37.824780 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:37.824786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:37.824789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:37.825062 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:37.825214 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:37.825869 (+ 655us) server_negotiation.cc:299] Negotiation successful
0923 20:15:37.826095 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:15:37.827368 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:37.809827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:37393 (local address 127.5.83.67:41211)
0923 20:15:37.810604 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:37.810641 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:37.812250 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:37.813014 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:37.813021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:37.813518 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:37.814051 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:37.814074 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.815131 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.815136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:37.816325 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:37.816336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.817413 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.817419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:37.817611 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:37.818327 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:37.818348 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:37.820766 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:37.823591 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:37.823600 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:37.823615 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:37.824018 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:37.824398 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:37.824404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:37.824408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:37.824672 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:37.825188 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:37.825196 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:37.825525 (+ 329us) client_negotiation.cc:769] Sending connection context
0923 20:15:37.825778 (+ 253us) client_negotiation.cc:240] Negotiation successful
0923 20:15:37.826296 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":333,"threads_started":1}
I20260923 20:15:37.828724 8984 heartbeater.cc:347] Connected to a master server at 127.5.83.126:37393
I20260923 20:15:37.829034 8984 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:37.829761 8984 heartbeater.cc:511] Master 127.5.83.126:37393 requested a full tablet report, sending...
I20260923 20:15:37.831152 8528 ts_manager.cc:194] Registered new tserver with Master: 357c8253f3434d21867cfcc263070471 (127.5.83.67:41819)
I20260923 20:15:37.832155 8528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:41211
I20260923 20:15:37.833297 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:15:37.841532 8717 heartbeater.cc:503] Master 127.5.83.126:37393 was elected leader, sending a full tablet report...
Sep 23 20:15:37 dist-test-slave-vl24 krb5kdc[8483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194537, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 20:15:37 dist-test-slave-vl24 krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194537, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:37.883862 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:37.869407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.5.83.126:37393)
0923 20:15:37.869686 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:37.869694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:37.870045 (+ 351us) server_negotiation.cc:407] Connection header received
0923 20:15:37.870275 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:37.870281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:37.870359 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:37.870471 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:37.871914 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.872749 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.873991 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.874429 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.874574 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:37.878338 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:37.878351 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:37.878354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:37.878387 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:37.881008 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:37.881710 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:37.881717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:37.881721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:37.881877 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:37.882301 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:37.882308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:37.882312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:37.882686 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:37.882951 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:37.883279 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 20:15:37.883534 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260923 20:15:37.884487 8528 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51598
I20260923 20:15:37.886503 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 8589
Sep 23 20:15:37 dist-test-slave-vl24 krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194537, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:37.902330 8995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:37.887607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.5.83.65:43139)
0923 20:15:37.888248 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:37.888255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:37.888280 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:15:37.888340 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:37.888346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:37.888618 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:37.888761 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:37.889937 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.890939 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.892150 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.892831 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.892930 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:37.896780 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:37.896791 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:37.896806 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:37.896838 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:37.899682 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:37.900457 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:37.900463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:37.900466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:37.900648 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:37.900990 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:37.900993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:37.900995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:37.901339 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:37.901552 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:37.901840 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 20:15:37.902037 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":337,"threads_started":1}
W20260923 20:15:37.903571 8712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54876
W20260923 20:15:37.904753 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 8589: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:15:37.904961 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 8589
I20260923 20:15:37.921237 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 8723
Sep 23 20:15:37 dist-test-slave-vl24 krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194537, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:15:37.937579 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:37.922091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.5.83.66:39041)
0923 20:15:37.922623 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:37.922630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:37.922657 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:15:37.922726 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:37.922732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:37.922939 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:37.923073 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:37.924434 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.925492 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.926826 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.927336 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.927432 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:37.931860 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:37.931870 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:37.931884 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:37.931924 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:37.934642 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:37.935386 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:37.935393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:37.935395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:37.935530 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:37.935900 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:37.935903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:37.935905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:37.936227 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:37.936448 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:37.936981 (+ 533us) server_negotiation.cc:299] Negotiation successful
0923 20:15:37.937188 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":277,"threads_started":1}
W20260923 20:15:37.938923 8846 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47868
W20260923 20:15:37.940186 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 8723: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:15:37.940363 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 8723
I20260923 20:15:37.956321 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 8856
Sep 23 20:15:37 dist-test-slave-vl24 krb5kdc[8483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194537, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:15:37.975845 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:37.957674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.5.83.67:41819)
0923 20:15:37.958485 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:37.958493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:37.958528 (+ 35us) server_negotiation.cc:407] Connection header received
0923 20:15:37.958596 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:37.958603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:37.958862 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:37.959077 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:37.960772 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.962075 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.963788 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:37.964466 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:37.964697 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:37.969628 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:37.969641 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:37.969658 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:37.969700 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:37.973191 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:37.973898 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:37.973905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:37.973907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:37.974072 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:37.974444 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:37.974448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:37.974450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:37.974821 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:37.975094 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:37.975344 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 20:15:37.975549 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":414,"threads_started":1}
W20260923 20:15:37.977221 8979 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43542
W20260923 20:15:37.978307 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 8856: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:15:37.978482 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 8856
I20260923 20:15:37.994482 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 8496
I20260923 20:15:37.995726 8558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:38.100291 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 8496
2026-09-23T20:15:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3616 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:15:38 dist-test-slave-vl24 krb5kdc[9001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:15:38 dist-test-slave-vl24 krb5kdc[9001](info): set up 2 sockets
Sep 23 20:15:38 dist-test-slave-vl24 krb5kdc[9001](info): commencing operation
krb5kdc: starting...
W20260923 20:15:39.198164 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:15:39 dist-test-slave-vl24 krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:15:39.401429 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36451
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:39.725484 9014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:39.725795 9014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:39.725967 9014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:39.733435 9014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:39.733542 9014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:39.733625 9014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:39.733736 9014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:39.733815 9014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:15:39.743906 9014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:39.745296 9014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:39.746594 9014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:39.758302 9019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:39.758356 9020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:39.758391 9022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:39.759160 9014 server_base.cc:1034] running on GCE node
I20260923 20:15:39.760059 9014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:39.761579 9014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:39.762830 9014 hybrid_clock.cc:648] HybridClock initialized: now 1790194539762791 us; error 50 us; skew 500 ppm
I20260923 20:15:39.763249 9014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:39 dist-test-slave-vl24 krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194539, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:39.766539 9014 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:15:39.768107 9014 webserver.cc:460] Webserver started at http://127.5.83.126:45331/ using document root <none> and password file <none>
I20260923 20:15:39.768869 9014 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:39.769012 9014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:39.769335 9014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:39.772506 9014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "c9befa1506b3495e846b33182dd35049"
format_stamp: "Formatted at 2026-09-23 20:15:39 on dist-test-slave-vl24"
server_key: "7d790ecf9ec6333091135626b6a41966"
server_key_iv: "c59a627011ad3021bc7de9c36b44caaa"
server_key_version: "encryptionkey@0"
I20260923 20:15:39.773376 9014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "c9befa1506b3495e846b33182dd35049"
format_stamp: "Formatted at 2026-09-23 20:15:39 on dist-test-slave-vl24"
server_key: "7d790ecf9ec6333091135626b6a41966"
server_key_iv: "c59a627011ad3021bc7de9c36b44caaa"
server_key_version: "encryptionkey@0"
I20260923 20:15:39.778703 9014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:15:39.782580 9029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:39.783965 9014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:15:39.784195 9014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "c9befa1506b3495e846b33182dd35049"
format_stamp: "Formatted at 2026-09-23 20:15:39 on dist-test-slave-vl24"
server_key: "7d790ecf9ec6333091135626b6a41966"
server_key_iv: "c59a627011ad3021bc7de9c36b44caaa"
server_key_version: "encryptionkey@0"
I20260923 20:15:39.784403 9014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:39.810943 9014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:39.814945 9014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:39.815251 9014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:39.833705 9014 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:36451
I20260923 20:15:39.833738 9080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:36451 every 8 connection(s)
I20260923 20:15:39.835155 9014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:39.838974 9081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:39.841799 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 9014
I20260923 20:15:39.842054 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:39.842741 5453 external_mini_cluster.cc:1428] Setting key 575324e5b4ec191abb397c0c9c8e334c
I20260923 20:15:39.852327 9081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049: Bootstrap starting.
I20260923 20:15:39.855953 9081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:39.857074 9081 log.cc:824] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:15:39 dist-test-slave-vl24 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:39.860354 9081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049: No bootstrap required, opened a new log
I20260923 20:15:39.868569 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:39.844917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.5.83.126:36451)
0923 20:15:39.845721 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:39.845736 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:39.845784 (+ 48us) server_negotiation.cc:407] Connection header received
0923 20:15:39.847929 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:39.847959 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:39.848546 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:39.848990 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:39.850483 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:39.852440 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:39.853842 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:39.854873 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:39.855067 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:39.859479 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:39.859492 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:39.859511 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:39.859566 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:39.863792 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:39.864497 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:39.864506 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:39.864515 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:39.864774 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:39.865134 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:39.865139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:39.865143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:39.865880 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:39.866176 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:39.866590 (+ 414us) server_negotiation.cc:299] Negotiation successful
0923 20:15:39.867068 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":324,"threads_started":1}
I20260923 20:15:39.871814 9081 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9befa1506b3495e846b33182dd35049" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36451 } }
I20260923 20:15:39.872167 9081 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:39.872252 9081 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9befa1506b3495e846b33182dd35049, State: Initialized, Role: FOLLOWER
I20260923 20:15:39.872866 9081 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9befa1506b3495e846b33182dd35049" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36451 } }
I20260923 20:15:39.873095 9081 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:39.873201 9081 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:39.873339 9081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:39.875691 9081 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9befa1506b3495e846b33182dd35049" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36451 } }
I20260923 20:15:39.876314 9081 leader_election.cc:302] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9befa1506b3495e846b33182dd35049; no voters:
I20260923 20:15:39.876972 9081 leader_election.cc:288] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:39.877100 9086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:39.877925 9086 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [term 1 LEADER]: Becoming Leader. State: Replica: c9befa1506b3495e846b33182dd35049, State: Running, Role: LEADER
I20260923 20:15:39.878513 9086 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9befa1506b3495e846b33182dd35049" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36451 } }
I20260923 20:15:39.879348 9081 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:39.882042 9087 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9befa1506b3495e846b33182dd35049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9befa1506b3495e846b33182dd35049" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36451 } } }
I20260923 20:15:39.882006 9088 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9befa1506b3495e846b33182dd35049. Latest consensus state: current_term: 1 leader_uuid: "c9befa1506b3495e846b33182dd35049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9befa1506b3495e846b33182dd35049" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36451 } } }
I20260923 20:15:39.882512 9087 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:39.882521 9088 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:39.883332 9095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:39.889385 9095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:39.890179 9103 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:15:39.897897 9095 catalog_manager.cc:1347] Generated new cluster ID: 323a18b5536e47b291e52db3da69df24
I20260923 20:15:39.898027 9095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:39.907814 9095 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:39.909243 9095 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:39.922207 9095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9befa1506b3495e846b33182dd35049: Generated new TSK 0
I20260923 20:15:39.923295 9095 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:15:39.984745 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36451
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:40.313897 9107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:40.314189 9107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:40.314284 9107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:40.321698 9107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:40.321811 9107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:40.321928 9107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:15:40.331804 9107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:40.333243 9107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:40.334520 9107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:40.347465 9112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:40.347502 9113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:40.347427 9115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:40.348505 9107 server_base.cc:1034] running on GCE node
I20260923 20:15:40.349236 9107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:40.350220 9107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:40.351480 9107 hybrid_clock.cc:648] HybridClock initialized: now 1790194540351422 us; error 44 us; skew 500 ppm
I20260923 20:15:40.351858 9107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:40 dist-test-slave-vl24 krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194540, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:40.355423 9107 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:15:40.357004 9107 webserver.cc:460] Webserver started at http://127.5.83.65:41669/ using document root <none> and password file <none>
I20260923 20:15:40.357744 9107 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:40.357903 9107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:40.358268 9107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:40.361522 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "bb4f657947374b4180b92d1aa5c0dcb9"
format_stamp: "Formatted at 2026-09-23 20:15:40 on dist-test-slave-vl24"
server_key: "f8b5121f3a6403d05f24fd3ad73176ae"
server_key_iv: "166b6b5572d3e2a856af08ceb6588b45"
server_key_version: "encryptionkey@0"
I20260923 20:15:40.362370 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "bb4f657947374b4180b92d1aa5c0dcb9"
format_stamp: "Formatted at 2026-09-23 20:15:40 on dist-test-slave-vl24"
server_key: "f8b5121f3a6403d05f24fd3ad73176ae"
server_key_iv: "166b6b5572d3e2a856af08ceb6588b45"
server_key_version: "encryptionkey@0"
I20260923 20:15:40.368040 9107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:15:40.371970 9122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:40.373574 9107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:40.373755 9107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "bb4f657947374b4180b92d1aa5c0dcb9"
format_stamp: "Formatted at 2026-09-23 20:15:40 on dist-test-slave-vl24"
server_key: "f8b5121f3a6403d05f24fd3ad73176ae"
server_key_iv: "166b6b5572d3e2a856af08ceb6588b45"
server_key_version: "encryptionkey@0"
I20260923 20:15:40.373973 9107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:40.416767 9107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:40.420732 9107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:40.421012 9107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:40.421993 9107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:40.423333 9107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:40.423430 9107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:40.423571 9107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:40.423642 9107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:40.456087 9107 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:45691
I20260923 20:15:40.456126 9234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:45691 every 8 connection(s)
I20260923 20:15:40.457482 9107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:15:40.465482 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 9107
I20260923 20:15:40.465700 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:15:40.466395 5453 external_mini_cluster.cc:1428] Setting key d29f3835104e29fa750ed710fd1b5c84
Sep 23 20:15:40 dist-test-slave-vl24 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194540, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:40.478750 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:40.460070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:51209 (local address 127.5.83.126:36451)
0923 20:15:40.460511 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:40.460515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:40.461562 (+ 1047us) server_negotiation.cc:407] Connection header received
0923 20:15:40.463395 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:40.463400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:40.463491 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:40.463607 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:40.465243 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:40.466109 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:40.468117 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:40.468661 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:40.468766 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:40.472971 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:40.472976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:40.472980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:40.473013 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:40.475591 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:40.476324 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:40.476334 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:40.476336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:40.476451 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:40.476925 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:40.476930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:40.476934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:40.477245 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:40.477406 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:40.478162 (+ 756us) server_negotiation.cc:299] Negotiation successful
0923 20:15:40.478384 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
I20260923 20:15:40.479836 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:40.460736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36451 (local address 127.5.83.65:51209)
0923 20:15:40.461404 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:40.461443 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:40.463128 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:40.463841 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:40.463850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:40.464380 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:40.465033 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:40.465060 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:40.466271 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:40.466278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:40.467929 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:40.467942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:40.468887 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:40.468895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:40.469206 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:40.469927 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:40.469962 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:40.472741 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:40.475739 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:40.475748 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:40.475767 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:40.476194 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:40.476589 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:40.476594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:40.476597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:40.476802 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:40.477376 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:40.477386 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:40.477728 (+ 342us) client_negotiation.cc:769] Sending connection context
0923 20:15:40.478041 (+ 313us) client_negotiation.cc:240] Negotiation successful
0923 20:15:40.478616 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":262,"threads_started":1}
I20260923 20:15:40.481628 9235 heartbeater.cc:347] Connected to a master server at 127.5.83.126:36451
I20260923 20:15:40.481945 9235 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:40.482759 9235 heartbeater.cc:511] Master 127.5.83.126:36451 requested a full tablet report, sending...
I20260923 20:15:40.485212 9046 ts_manager.cc:194] Registered new tserver with Master: bb4f657947374b4180b92d1aa5c0dcb9 (127.5.83.65:45691)
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
I20260923 20:15:40.487241 9046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:51209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:15:40.513849 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36451
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:40.837738 9241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:40.838106 9241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:40.838255 9241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:40.849912 9241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:40.850059 9241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:40.850241 9241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:15:40.862648 9241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:40.863945 9241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:40.865355 9241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:40.878906 9249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:40.878872 9247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:40.879293 9246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:40.879688 9241 server_base.cc:1034] running on GCE node
I20260923 20:15:40.880334 9241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:40.881433 9241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:40.882624 9241 hybrid_clock.cc:648] HybridClock initialized: now 1790194540882582 us; error 38 us; skew 500 ppm
I20260923 20:15:40.882948 9241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:40 dist-test-slave-vl24 krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194540, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:40.886269 9241 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:15:40.887785 9241 webserver.cc:460] Webserver started at http://127.5.83.66:41753/ using document root <none> and password file <none>
I20260923 20:15:40.888468 9241 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:40.888643 9241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:40.888916 9241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:40.891755 9241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "25e0797b36bf411996b9bae46d1173a3"
format_stamp: "Formatted at 2026-09-23 20:15:40 on dist-test-slave-vl24"
server_key: "5891139d68f54187e78f70150f92acd2"
server_key_iv: "dd7b55aabe211c8b63195cc401ae39fb"
server_key_version: "encryptionkey@0"
I20260923 20:15:40.892638 9241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "25e0797b36bf411996b9bae46d1173a3"
format_stamp: "Formatted at 2026-09-23 20:15:40 on dist-test-slave-vl24"
server_key: "5891139d68f54187e78f70150f92acd2"
server_key_iv: "dd7b55aabe211c8b63195cc401ae39fb"
server_key_version: "encryptionkey@0"
I20260923 20:15:40.897874 9241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:15:40.901405 9256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:40.902842 9241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:15:40.903023 9241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "25e0797b36bf411996b9bae46d1173a3"
format_stamp: "Formatted at 2026-09-23 20:15:40 on dist-test-slave-vl24"
server_key: "5891139d68f54187e78f70150f92acd2"
server_key_iv: "dd7b55aabe211c8b63195cc401ae39fb"
server_key_version: "encryptionkey@0"
I20260923 20:15:40.903251 9241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:40.930328 9241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:40.934878 9241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:40.935184 9241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:40.936139 9241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:40.937598 9241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:40.937703 9241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:40.937848 9241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:40.937920 9241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:40.971463 9241 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:39239
I20260923 20:15:40.971508 9368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:39239 every 8 connection(s)
I20260923 20:15:40.972859 9241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:15:40.973778 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 9241
I20260923 20:15:40.973938 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:15:40.974479 5453 external_mini_cluster.cc:1428] Setting key 72bb39b742df6badcda55a3f25b886f8
Sep 23 20:15:40 dist-test-slave-vl24 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194540, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
I20260923 20:15:40.995370 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:40.975610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:48193 (local address 127.5.83.126:36451)
0923 20:15:40.975787 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:40.975791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:40.977312 (+ 1521us) server_negotiation.cc:407] Connection header received
0923 20:15:40.979361 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:40.979369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:40.979484 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:40.979608 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:40.981321 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:40.982526 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:40.984518 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:40.985017 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:40.985124 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:40.989551 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:40.989556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:40.989560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:40.989593 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:40.992158 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:40.993020 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:40.993025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:40.993027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:40.993172 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:40.993609 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:40.993616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:40.993624 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:40.993890 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:40.994055 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:40.994773 (+ 718us) server_negotiation.cc:299] Negotiation successful
0923 20:15:40.995007 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:15:40.996410 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:40.976248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36451 (local address 127.5.83.66:48193)
0923 20:15:40.977136 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:40.977182 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:40.979093 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:40.979876 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:40.979884 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:40.980403 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:40.981138 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:40.981157 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:40.982716 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:40.982721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:40.984341 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:40.984363 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:40.985247 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:40.985256 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:40.985592 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:40.986353 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:40.986389 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:40.989303 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:40.992315 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:40.992325 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:40.992344 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:40.992881 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:40.993290 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:40.993297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:40.993302 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:40.993488 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:40.994021 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:40.994029 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:40.994389 (+ 360us) client_negotiation.cc:769] Sending connection context
0923 20:15:40.994663 (+ 274us) client_negotiation.cc:240] Negotiation successful
0923 20:15:40.995208 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":405,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
I20260923 20:15:40.997900 9369 heartbeater.cc:347] Connected to a master server at 127.5.83.126:36451
I20260923 20:15:40.998260 9369 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:40.999179 9369 heartbeater.cc:511] Master 127.5.83.126:36451 requested a full tablet report, sending...
I20260923 20:15:41.000952 9046 ts_manager.cc:194] Registered new tserver with Master: 25e0797b36bf411996b9bae46d1173a3 (127.5.83.66:39239)
I20260923 20:15:41.002224 9046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:48193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:15:41.023437 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36451
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:41.363909 9374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:41.364283 9374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:41.364476 9374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:41.371953 9374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:41.372110 9374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:41.372293 9374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:15:41.383237 9374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:41.384708 9374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:41.386050 9374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:41.400177 9379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:41.400204 9380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:41.400249 9382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:41.400911 9374 server_base.cc:1034] running on GCE node
I20260923 20:15:41.401680 9374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:41.402688 9374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:41.403913 9374 hybrid_clock.cc:648] HybridClock initialized: now 1790194541403876 us; error 53 us; skew 500 ppm
I20260923 20:15:41.404238 9374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:41 dist-test-slave-vl24 krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194541, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:41.419504 9374 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:15:41.421121 9374 webserver.cc:460] Webserver started at http://127.5.83.67:44199/ using document root <none> and password file <none>
I20260923 20:15:41.421779 9374 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:41.421882 9374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:41.422151 9374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:41.425064 9374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "f4d49b7ce72d473d855d90af3b1d267f"
format_stamp: "Formatted at 2026-09-23 20:15:41 on dist-test-slave-vl24"
server_key: "7edda5089e6fa5cb8bbfac0cb163d166"
server_key_iv: "e3cf234b498fe6810e8ba5e7e74e6ccc"
server_key_version: "encryptionkey@0"
I20260923 20:15:41.425778 9374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "f4d49b7ce72d473d855d90af3b1d267f"
format_stamp: "Formatted at 2026-09-23 20:15:41 on dist-test-slave-vl24"
server_key: "7edda5089e6fa5cb8bbfac0cb163d166"
server_key_iv: "e3cf234b498fe6810e8ba5e7e74e6ccc"
server_key_version: "encryptionkey@0"
I20260923 20:15:41.430790 9374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:15:41.434094 9389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:41.435369 9374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:15:41.435524 9374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "f4d49b7ce72d473d855d90af3b1d267f"
format_stamp: "Formatted at 2026-09-23 20:15:41 on dist-test-slave-vl24"
server_key: "7edda5089e6fa5cb8bbfac0cb163d166"
server_key_iv: "e3cf234b498fe6810e8ba5e7e74e6ccc"
server_key_version: "encryptionkey@0"
I20260923 20:15:41.435691 9374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:41.454576 9374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:41.459281 9374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:41.459676 9374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:41.460748 9374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:41.462178 9374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:41.462291 9374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:41.462455 9374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:41.462533 9374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:41.490542 9235 heartbeater.cc:503] Master 127.5.83.126:36451 was elected leader, sending a full tablet report...
I20260923 20:15:41.496886 9374 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:41915
I20260923 20:15:41.496979 9501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:41915 every 8 connection(s)
I20260923 20:15:41.498180 9374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:15:41.504159 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 9374
I20260923 20:15:41.504351 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:15:41.504964 5453 external_mini_cluster.cc:1428] Setting key 54f78f22b4458fe1a19586269b49fb4c
Sep 23 20:15:41 dist-test-slave-vl24 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194541, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:41.519219 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:41.500667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:52777 (local address 127.5.83.126:36451)
0923 20:15:41.501112 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:41.501116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:41.501977 (+ 861us) server_negotiation.cc:407] Connection header received
0923 20:15:41.503837 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:41.503842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:41.503911 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:41.504040 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:41.506190 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.507327 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.509222 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.509618 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.509721 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:41.513093 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:41.513098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:41.513101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:41.513137 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:41.516178 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:41.516957 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:41.516961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:41.516964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:41.517124 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:41.517518 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:41.517522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:41.517524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:41.517836 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:41.517951 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:41.518630 (+ 679us) server_negotiation.cc:299] Negotiation successful
0923 20:15:41.518798 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":237,"threads_started":1}
I20260923 20:15:41.520503 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:41.501130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36451 (local address 127.5.83.67:52777)
0923 20:15:41.501817 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:41.501864 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:41.503531 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:41.504343 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:41.504357 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:41.505188 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:41.505957 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:41.505984 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.507469 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.507472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:41.509092 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:41.509105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.509805 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.509810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:41.509999 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:41.510646 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:41.510671 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:41.512859 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:41.516302 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:41.516309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:41.516321 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:41.516830 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:41.517230 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:41.517233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:41.517235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:41.517408 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:41.517945 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:41.517950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:41.518269 (+ 319us) client_negotiation.cc:769] Sending connection context
0923 20:15:41.518531 (+ 262us) client_negotiation.cc:240] Negotiation successful
0923 20:15:41.519022 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":263,"threads_started":1}
I20260923 20:15:41.521891 9502 heartbeater.cc:347] Connected to a master server at 127.5.83.126:36451
I20260923 20:15:41.522203 9502 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:41.522895 9502 heartbeater.cc:511] Master 127.5.83.126:36451 requested a full tablet report, sending...
I20260923 20:15:41.524303 9046 ts_manager.cc:194] Registered new tserver with Master: f4d49b7ce72d473d855d90af3b1d267f (127.5.83.67:41915)
I20260923 20:15:41.525146 9046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:52777
I20260923 20:15:41.532940 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:15:41 dist-test-slave-vl24 krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:15:41 dist-test-slave-vl24 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:41.585587 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:41.569754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.5.83.65:45691)
0923 20:15:41.570633 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:41.570643 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:41.570679 (+ 36us) server_negotiation.cc:407] Connection header received
0923 20:15:41.570781 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:41.570788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:41.571010 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:41.571169 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:41.572701 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.573736 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.575021 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.575594 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.575711 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:41.580078 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:41.580089 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:41.580118 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:41.580156 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:41.583002 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:41.583736 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:41.583743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:41.583745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:41.583872 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:41.584222 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:41.584226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:41.584228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:41.584633 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:41.584865 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:41.585118 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 20:15:41.585315 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":503,"threads_started":1}
Sep 23 20:15:41 dist-test-slave-vl24 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:41.603549 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:41.590120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.5.83.126:36451)
0923 20:15:41.590348 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:41.590352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:41.590452 (+ 100us) server_negotiation.cc:407] Connection header received
0923 20:15:41.590591 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:41.590595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:41.590663 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:41.590741 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:41.591845 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.592646 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.593921 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.594357 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.594468 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:41.598504 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:41.598510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:41.598513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:41.598551 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:41.601270 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:41.601899 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:41.601903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:41.601905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:41.602051 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:41.602382 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:41.602384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:41.602386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:41.602703 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:41.602889 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:41.603156 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 20:15:41.603320 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260923 20:15:41.604214 9046 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36432
I20260923 20:15:41.605345 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 9107
I20260923 20:15:41.619249 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:41.606208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.5.83.65:45691)
0923 20:15:41.606400 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:41.606405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:41.606710 (+ 305us) server_negotiation.cc:407] Connection header received
0923 20:15:41.606936 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:41.606940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:41.607076 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:41.607182 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:41.608666 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.609375 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.611189 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.611642 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.611772 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:41.614246 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:41.614251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:41.614255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:41.614293 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:41.617022 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:41.617678 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:41.617682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:41.617684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:41.617810 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:41.618134 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:41.618137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:41.618139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:41.618388 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:41.618561 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:41.618835 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 20:15:41.619010 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:15:41.619992 9230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:41.702518 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 9107
I20260923 20:15:41.719108 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 9241
Sep 23 20:15:41 dist-test-slave-vl24 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:15:41.734900 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:41.719961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.5.83.66:39239)
0923 20:15:41.720493 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:41.720500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:41.720576 (+ 76us) server_negotiation.cc:407] Connection header received
0923 20:15:41.720648 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:41.720654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:41.720864 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:41.720991 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:41.722210 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.723186 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.724507 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.725172 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.725273 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:41.729428 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:41.729438 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:41.729453 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:41.729493 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:41.732270 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:41.732945 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:41.732952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:41.732954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:41.733081 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:41.733406 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:41.733409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:41.733410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:41.733790 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:41.734009 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:41.734361 (+ 352us) server_negotiation.cc:299] Negotiation successful
0923 20:15:41.734582 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":289,"threads_started":1}
I20260923 20:15:41.736285 9364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:41.819651 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 9241
I20260923 20:15:41.836982 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 9374
Sep 23 20:15:41 dist-test-slave-vl24 krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:15:41.854338 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:41.837995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.5.83.67:41915)
0923 20:15:41.838667 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:41.838677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:41.838713 (+ 36us) server_negotiation.cc:407] Connection header received
0923 20:15:41.838786 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:41.838794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:41.839085 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:41.839259 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:41.840802 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.842006 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.843321 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:41.843860 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:41.843956 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:41.848726 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:41.848738 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:41.848753 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:41.848789 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:41.851703 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:41.852351 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:41.852358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:41.852361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:41.852477 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:41.852864 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:41.852867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:41.852870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:41.853239 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:41.853468 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:41.853729 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 20:15:41.853923 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":323,"threads_started":1}
I20260923 20:15:41.855688 9497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:41.943347 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 9374
I20260923 20:15:41.959510 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 9014
I20260923 20:15:41.960764 9076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:42.065196 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 9014
2026-09-23T20:15:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3966 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:15:42 dist-test-slave-vl24 krb5kdc[9522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:15:42 dist-test-slave-vl24 krb5kdc[9522](info): set up 2 sockets
Sep 23 20:15:42 dist-test-slave-vl24 krb5kdc[9522](info): commencing operation
krb5kdc: starting...
W20260923 20:15:43.154763 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:15:43 dist-test-slave-vl24 krb5kdc[9522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:15:43.362150 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:38717
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:38333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:38717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:43.707969 9535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:43.708292 9535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:43.708451 9535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:43.715795 9535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:43.715926 9535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:43.716012 9535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:43.716111 9535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:43.716200 9535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:15:43.726275 9535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:38333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:38717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:38717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:43.727581 9535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:43.728957 9535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:43.740475 9541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:43.740460 9543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:43.740799 9540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:43.741324 9535 server_base.cc:1034] running on GCE node
I20260923 20:15:43.742069 9535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:43.743475 9535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:43.744645 9535 hybrid_clock.cc:648] HybridClock initialized: now 1790194543744596 us; error 27 us; skew 500 ppm
I20260923 20:15:43.745046 9535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:43 dist-test-slave-vl24 krb5kdc[9522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194543, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:43.748129 9535 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:15:43.749322 9535 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:43.749439 9535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:43.749794 9535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:43.752827 9535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "5bad87a8a9e24c898a96270d37883fd0"
format_stamp: "Formatted at 2026-09-23 20:15:43 on dist-test-slave-vl24"
server_key: "23524aa639b009be1bf6506d3ebcc950"
server_key_iv: "6cc8cf1e6e4531d43c590245ee0dd899"
server_key_version: "encryptionkey@0"
I20260923 20:15:43.753631 9535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "5bad87a8a9e24c898a96270d37883fd0"
format_stamp: "Formatted at 2026-09-23 20:15:43 on dist-test-slave-vl24"
server_key: "23524aa639b009be1bf6506d3ebcc950"
server_key_iv: "6cc8cf1e6e4531d43c590245ee0dd899"
server_key_version: "encryptionkey@0"
I20260923 20:15:43.758653 9535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:15:43.762089 9549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:43.763383 9535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:15:43.763561 9535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "5bad87a8a9e24c898a96270d37883fd0"
format_stamp: "Formatted at 2026-09-23 20:15:43 on dist-test-slave-vl24"
server_key: "23524aa639b009be1bf6506d3ebcc950"
server_key_iv: "6cc8cf1e6e4531d43c590245ee0dd899"
server_key_version: "encryptionkey@0"
I20260923 20:15:43.763824 9535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:43.789804 9535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:43.794445 9535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:43.794734 9535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:43.812507 9535 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:38717
I20260923 20:15:43.812554 9600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:38717 every 8 connection(s)
I20260923 20:15:43.813488 9535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:43.817123 9601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:43.821599 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 9535
I20260923 20:15:43.821820 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:43.822443 5453 external_mini_cluster.cc:1428] Setting key 0978608c139a239431dc7a471496e37a
I20260923 20:15:43.829504 9601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0: Bootstrap starting.
I20260923 20:15:43.833165 9601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:43.834368 9601 log.cc:824] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:43.837340 9601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0: No bootstrap required, opened a new log
Sep 23 20:15:43 dist-test-slave-vl24 krb5kdc[9522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:43.848229 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:43.824412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.5.83.126:38717)
0923 20:15:43.825308 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:43.825322 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:43.825371 (+ 49us) server_negotiation.cc:407] Connection header received
0923 20:15:43.827904 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:43.827931 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:43.828424 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:43.828882 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:43.830312 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:43.832380 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:43.833778 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:43.834839 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:43.835094 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:43.839374 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:43.839384 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:43.839404 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:43.839452 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:43.843536 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:43.844192 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:43.844201 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:43.844210 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:43.844438 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:43.844845 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:43.844850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:43.844854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:43.845602 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:43.845935 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:43.846310 (+ 375us) server_negotiation.cc:299] Negotiation successful
0923 20:15:43.846807 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":359,"threads_started":1}
I20260923 20:15:43.848279 9601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bad87a8a9e24c898a96270d37883fd0" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38717 } }
I20260923 20:15:43.848734 9601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:43.848852 9601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bad87a8a9e24c898a96270d37883fd0, State: Initialized, Role: FOLLOWER
I20260923 20:15:43.849535 9601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bad87a8a9e24c898a96270d37883fd0" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38717 } }
I20260923 20:15:43.849820 9601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:43.849965 9601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:43.850147 9601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:43.852692 9601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bad87a8a9e24c898a96270d37883fd0" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38717 } }
I20260923 20:15:43.853190 9601 leader_election.cc:302] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bad87a8a9e24c898a96270d37883fd0; no voters:
I20260923 20:15:43.853847 9601 leader_election.cc:288] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:43.853953 9606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:43.854651 9606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [term 1 LEADER]: Becoming Leader. State: Replica: 5bad87a8a9e24c898a96270d37883fd0, State: Running, Role: LEADER
I20260923 20:15:43.855278 9606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bad87a8a9e24c898a96270d37883fd0" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38717 } }
I20260923 20:15:43.856122 9601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:43.858279 9607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bad87a8a9e24c898a96270d37883fd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bad87a8a9e24c898a96270d37883fd0" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38717 } } }
I20260923 20:15:43.858702 9607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:43.858892 9608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bad87a8a9e24c898a96270d37883fd0. Latest consensus state: current_term: 1 leader_uuid: "5bad87a8a9e24c898a96270d37883fd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bad87a8a9e24c898a96270d37883fd0" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38717 } } }
I20260923 20:15:43.861032 9608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:43.867538 9611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:43.873317 9611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:43.881544 9611 catalog_manager.cc:1347] Generated new cluster ID: b3a708e750ab41e59d838a828c792d7a
I20260923 20:15:43.881668 9611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:43.904561 9611 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:43.905982 9611 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:43.916967 9611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0: Generated new TSK 0
I20260923 20:15:43.918031 9611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:15:43.972400 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:38717
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:38333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:44.306528 9627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:44.306815 9627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:44.306914 9627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:44.314416 9627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:44.314538 9627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:44.314656 9627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:15:44.324862 9627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:38333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:38717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:44.326224 9627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:44.327498 9627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:44.340605 9632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:44.340490 9635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:44.340488 9633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:44.341684 9627 server_base.cc:1034] running on GCE node
I20260923 20:15:44.342386 9627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:44.343475 9627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:44.344748 9627 hybrid_clock.cc:648] HybridClock initialized: now 1790194544344666 us; error 64 us; skew 500 ppm
I20260923 20:15:44.345126 9627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:44 dist-test-slave-vl24 krb5kdc[9522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194544, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:44.348744 9627 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:15:44.350080 9627 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:44.350191 9627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:44.350510 9627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:44.353926 9627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "89dd3c3fd9754d86998371040987392b"
format_stamp: "Formatted at 2026-09-23 20:15:44 on dist-test-slave-vl24"
server_key: "82ee28c09f48510c6787728df4e934c3"
server_key_iv: "fd18318b893f59eac39038c362850ef3"
server_key_version: "encryptionkey@0"
I20260923 20:15:44.354923 9627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "89dd3c3fd9754d86998371040987392b"
format_stamp: "Formatted at 2026-09-23 20:15:44 on dist-test-slave-vl24"
server_key: "82ee28c09f48510c6787728df4e934c3"
server_key_iv: "fd18318b893f59eac39038c362850ef3"
server_key_version: "encryptionkey@0"
I20260923 20:15:44.360824 9627 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260923 20:15:44.364926 9641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:44.366365 9627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:15:44.366577 9627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "89dd3c3fd9754d86998371040987392b"
format_stamp: "Formatted at 2026-09-23 20:15:44 on dist-test-slave-vl24"
server_key: "82ee28c09f48510c6787728df4e934c3"
server_key_iv: "fd18318b893f59eac39038c362850ef3"
server_key_version: "encryptionkey@0"
I20260923 20:15:44.366824 9627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:44.388682 9627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:44.392769 9627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:44.393003 9627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:44.393946 9627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:44.395263 9627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:44.395349 9627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:44.395432 9627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:44.395486 9627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:44.429311 9627 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:44979
I20260923 20:15:44.429397 9753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:44979 every 8 connection(s)
I20260923 20:15:44.430203 9627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:15:44.432643 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 9627
I20260923 20:15:44.432855 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:15:44.433714 5453 external_mini_cluster.cc:1428] Setting key a8c402eab5627b264dad58a7dec31ee9
Sep 23 20:15:44 dist-test-slave-vl24 krb5kdc[9522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194544, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:44.453068 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:44.432652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:44059 (local address 127.5.83.126:38717)
0923 20:15:44.433160 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:44.433168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:44.434249 (+ 1081us) server_negotiation.cc:407] Connection header received
0923 20:15:44.436642 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:44.436646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:44.436725 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:44.436815 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:44.438937 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:44.439656 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:44.442249 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:44.442679 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:44.442790 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:44.447155 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:44.447160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:44.447163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:44.447197 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:44.449897 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:44.450696 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:44.450708 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:44.450711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:44.450825 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:44.451263 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:44.451268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:44.451271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:44.451549 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:44.451700 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:44.452428 (+ 728us) server_negotiation.cc:299] Negotiation successful
0923 20:15:44.452692 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":219,"threads_started":1}
I20260923 20:15:44.454020 9757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:44.433170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:38717 (local address 127.5.83.65:44059)
0923 20:15:44.434034 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:44.434091 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:44.436288 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:44.437048 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:44.437064 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:44.437902 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:44.438750 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:44.438779 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:44.439828 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:44.439837 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:44.442067 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:44.442086 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:44.442929 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:44.442938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:44.443279 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:44.444066 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:44.444101 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:44.446900 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:44.450093 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:44.450103 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:44.450123 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:44.450565 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:44.450937 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:44.450943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:44.450947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:44.451141 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:44.451693 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:44.451702 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:44.452044 (+ 342us) client_negotiation.cc:769] Sending connection context
0923 20:15:44.452312 (+ 268us) client_negotiation.cc:240] Negotiation successful
0923 20:15:44.452870 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":329,"threads_started":1}
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
I20260923 20:15:44.455927 9754 heartbeater.cc:347] Connected to a master server at 127.5.83.126:38717
I20260923 20:15:44.456245 9754 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:44.457147 9754 heartbeater.cc:511] Master 127.5.83.126:38717 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
I20260923 20:15:44.459409 9566 ts_manager.cc:194] Registered new tserver with Master: 89dd3c3fd9754d86998371040987392b (127.5.83.65:44979)
I20260923 20:15:44.460906 9566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:44059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:15:44.481967 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:38717
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:38333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:44.802554 9760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:44.802830 9760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:44.803004 9760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:44.810249 9760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:44.810384 9760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:44.810575 9760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:15:44.825528 9760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:38333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:38717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:44.827255 9760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:44.828930 9760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:44.843211 9765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:44.843165 9766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:44.843235 9768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:44.843904 9760 server_base.cc:1034] running on GCE node
I20260923 20:15:44.844439 9760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:44.845381 9760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:44.846573 9760 hybrid_clock.cc:648] HybridClock initialized: now 1790194544846524 us; error 51 us; skew 500 ppm
I20260923 20:15:44.846884 9760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:44 dist-test-slave-vl24 krb5kdc[9522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194544, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:44.849890 9760 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:15:44.851022 9760 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:44.851159 9760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:44.851472 9760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:44.854391 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "e05e5b675a4a490fb6314d191ab8d03c"
format_stamp: "Formatted at 2026-09-23 20:15:44 on dist-test-slave-vl24"
server_key: "12e854ee75b3ed7fbb17ec644a203986"
server_key_iv: "d5d8af11f0ebfdf0dc7d91376d1f05fb"
server_key_version: "encryptionkey@0"
I20260923 20:15:44.855191 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "e05e5b675a4a490fb6314d191ab8d03c"
format_stamp: "Formatted at 2026-09-23 20:15:44 on dist-test-slave-vl24"
server_key: "12e854ee75b3ed7fbb17ec644a203986"
server_key_iv: "d5d8af11f0ebfdf0dc7d91376d1f05fb"
server_key_version: "encryptionkey@0"
I20260923 20:15:44.860208 9760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 20:15:44.863721 9774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:44.865202 9760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:44.865339 9760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "e05e5b675a4a490fb6314d191ab8d03c"
format_stamp: "Formatted at 2026-09-23 20:15:44 on dist-test-slave-vl24"
server_key: "12e854ee75b3ed7fbb17ec644a203986"
server_key_iv: "d5d8af11f0ebfdf0dc7d91376d1f05fb"
server_key_version: "encryptionkey@0"
I20260923 20:15:44.865528 9760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:44.879340 9760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:44.883137 9760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:44.883358 9760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:44.884215 9760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:44.885532 9760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:44.885622 9760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:44.885699 9760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:44.885754 9760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:44.918291 9760 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:46273
I20260923 20:15:44.918347 9886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:46273 every 8 connection(s)
I20260923 20:15:44.919158 9760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:15:44.921943 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 9760
I20260923 20:15:44.922132 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:15:44.922719 5453 external_mini_cluster.cc:1428] Setting key 38c27ec45f99c755913dc64e600a13ac
Sep 23 20:15:44 dist-test-slave-vl24 krb5kdc[9522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194544, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:44.941187 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:44.921787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:51529 (local address 127.5.83.126:38717)
0923 20:15:44.922033 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:44.922041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:44.923374 (+ 1333us) server_negotiation.cc:407] Connection header received
0923 20:15:44.925452 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:44.925461 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:44.925561 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:44.925680 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:44.927136 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:44.928247 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:44.929829 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:44.930622 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:44.930759 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:44.935273 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:44.935279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:44.935282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:44.935314 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:44.937842 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:44.938829 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:44.938832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:44.938835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:44.938951 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:44.939446 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:44.939450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:44.939454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:44.939738 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:44.939844 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:44.940802 (+ 958us) server_negotiation.cc:299] Negotiation successful
0923 20:15:44.940959 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:15:44.942966 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:44.922358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:38717 (local address 127.5.83.66:51529)
0923 20:15:44.923191 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:44.923231 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:44.925177 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:44.925875 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:44.925884 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:44.926393 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:44.926950 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:44.926967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:44.928383 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:44.928386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:44.929684 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:44.929696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:44.930877 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:44.930885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:44.931187 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:44.931989 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:44.932029 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:44.935044 (+ 3015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:44.937985 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:44.937997 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:44.938016 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:44.938712 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:44.939059 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:44.939068 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:44.939074 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:44.939315 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:44.939868 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:44.939876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:44.940360 (+ 484us) client_negotiation.cc:769] Sending connection context
0923 20:15:44.940722 (+ 362us) client_negotiation.cc:240] Negotiation successful
0923 20:15:44.941414 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":387,"threads_started":1}
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
I20260923 20:15:44.944573 9887 heartbeater.cc:347] Connected to a master server at 127.5.83.126:38717
I20260923 20:15:44.944881 9887 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:44.945498 9887 heartbeater.cc:511] Master 127.5.83.126:38717 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
I20260923 20:15:44.947109 9566 ts_manager.cc:194] Registered new tserver with Master: e05e5b675a4a490fb6314d191ab8d03c (127.5.83.66:46273)
I20260923 20:15:44.948390 9566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:51529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:15:44.972114 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:38717
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:38333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:45.299556 9892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:45.299836 9892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:45.300051 9892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:45.307550 9892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:45.307749 9892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:45.307969 9892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:15:45.319576 9892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:38333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:38717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:45.321557 9892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:45.323520 9892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:45.339481 9898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:45.339525 9897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:45.339891 9900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:45.340360 9892 server_base.cc:1034] running on GCE node
I20260923 20:15:45.340971 9892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:45.341883 9892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:45.343132 9892 hybrid_clock.cc:648] HybridClock initialized: now 1790194545343080 us; error 60 us; skew 500 ppm
I20260923 20:15:45.343466 9892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:45 dist-test-slave-vl24 krb5kdc[9522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194545, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:45.346824 9892 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:15:45.347946 9892 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:45.348042 9892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:45.348312 9892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:45.351221 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "989e4acac787450789dfb62005bcb142"
format_stamp: "Formatted at 2026-09-23 20:15:45 on dist-test-slave-vl24"
server_key: "94d7e1824532256103adece7edf759d0"
server_key_iv: "319ffb1edb5265e097f3532766a9eaa1"
server_key_version: "encryptionkey@0"
I20260923 20:15:45.351953 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "989e4acac787450789dfb62005bcb142"
format_stamp: "Formatted at 2026-09-23 20:15:45 on dist-test-slave-vl24"
server_key: "94d7e1824532256103adece7edf759d0"
server_key_iv: "319ffb1edb5265e097f3532766a9eaa1"
server_key_version: "encryptionkey@0"
I20260923 20:15:45.357204 9892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260923 20:15:45.360517 9906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:45.361882 9892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:15:45.362023 9892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "989e4acac787450789dfb62005bcb142"
format_stamp: "Formatted at 2026-09-23 20:15:45 on dist-test-slave-vl24"
server_key: "94d7e1824532256103adece7edf759d0"
server_key_iv: "319ffb1edb5265e097f3532766a9eaa1"
server_key_version: "encryptionkey@0"
I20260923 20:15:45.362191 9892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:45.385711 9892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:45.389667 9892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:45.389894 9892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:45.390913 9892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:45.392231 9892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:45.392342 9892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:45.392465 9892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:45.392557 9892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:45.426960 9892 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:33695
I20260923 20:15:45.427053 10018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:33695 every 8 connection(s)
I20260923 20:15:45.427794 9892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:15:45.431173 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 9892
I20260923 20:15:45.431348 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:15:45.431972 5453 external_mini_cluster.cc:1428] Setting key befdcba86f180f4b2987c6cdc7dd73fa
Sep 23 20:15:45 dist-test-slave-vl24 krb5kdc[9522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194545, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:45.448832 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.430578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:36119 (local address 127.5.83.126:38717)
0923 20:15:45.430796 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:45.430802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:45.432211 (+ 1409us) server_negotiation.cc:407] Connection header received
0923 20:15:45.434728 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:45.434733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:45.434856 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:45.434968 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:45.436453 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.437260 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.438737 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.439336 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.439460 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.442881 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:45.442888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:45.442892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:45.442942 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:45.445605 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:45.446429 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:45.446435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:45.446439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:45.446633 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:45.447145 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:45.447150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:45.447156 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:45.447570 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:45.447712 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:45.448360 (+ 648us) server_negotiation.cc:299] Negotiation successful
0923 20:15:45.448518 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:15:45.449812 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.431169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:38717 (local address 127.5.83.67:36119)
0923 20:15:45.432035 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:45.432085 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:45.434459 (+ 2374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:45.435241 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:45.435249 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:45.435736 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:45.436283 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:45.436300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.437387 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.437391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:45.438610 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:45.438623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.439543 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.439553 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:45.439888 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.440507 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:45.440575 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:45.442686 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:45.445737 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:45.445744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:45.445755 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:45.446289 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:45.446734 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:45.446740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:45.446744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:45.446918 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:45.447706 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:45.447712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:45.448044 (+ 332us) client_negotiation.cc:769] Sending connection context
0923 20:15:45.448268 (+ 224us) client_negotiation.cc:240] Negotiation successful
0923 20:15:45.448736 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":364,"threads_started":1}
I20260923 20:15:45.451241 10019 heartbeater.cc:347] Connected to a master server at 127.5.83.126:38717
I20260923 20:15:45.451515 10019 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:45.452119 10019 heartbeater.cc:511] Master 127.5.83.126:38717 requested a full tablet report, sending...
I20260923 20:15:45.453557 9566 ts_manager.cc:194] Registered new tserver with Master: 989e4acac787450789dfb62005bcb142 (127.5.83.67:33695)
I20260923 20:15:45.454401 9566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:36119
I20260923 20:15:45.459350 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:15:45.464092 9754 heartbeater.cc:503] Master 127.5.83.126:38717 was elected leader, sending a full tablet report...
I20260923 20:15:45.479365 9756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.463717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.5.83.126:38717)
0923 20:15:45.463889 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:45.463909 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:45.464402 (+ 493us) server_negotiation.cc:407] Connection header received
0923 20:15:45.464767 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:45.464770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:45.464887 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:45.465006 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:45.466764 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.467536 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.468993 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.469506 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.469635 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.472004 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:45.472009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:45.472013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:45.472042 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:45.475152 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:45.476370 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:45.476378 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:45.476382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:45.476753 (+ 371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:45.477271 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:45.477278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:45.477284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:45.477787 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:45.478039 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:45.478691 (+ 652us) server_negotiation.cc:299] Negotiation successful
0923 20:15:45.478944 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:15:45.490218 9566 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:15:45.504063 9566 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:15:45.543394 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.518685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44979 (local address 127.0.0.1:55642)
0923 20:15:45.532059 (+ 13374us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:45.532084 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:45.532259 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:45.533437 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:45.533448 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:45.533506 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:45.533924 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:45.533936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.538732 (+ 4796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.538737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:45.540992 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:45.541008 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.542389 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.542395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:45.542590 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.542648 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 20:15:45.542773 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 20:15:45.542992 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13210,"thread_start_us":13500,"threads_started":1}
I20260923 20:15:45.544139 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.518759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.5.83.65:44979)
0923 20:15:45.531246 (+ 12487us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:45.531261 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:45.532435 (+ 1174us) server_negotiation.cc:407] Connection header received
0923 20:15:45.532597 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:45.532609 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:45.532927 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:45.533129 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:45.534140 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.536967 (+ 2827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.541175 (+ 4208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.542200 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.542353 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.542590 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:45.543637 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0923 20:15:45.543810 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12216,"thread_start_us":12531,"threads_started":1}
I20260923 20:15:45.548306 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.518267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:46273 (local address 127.0.0.1:34698)
0923 20:15:45.531752 (+ 13485us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:45.531811 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:45.532033 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:45.538004 (+ 5971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:45.538007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:45.538027 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:45.538308 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:45.538314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.540259 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.540267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:45.542887 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:45.542910 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.547174 (+ 4264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.547183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:45.547417 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.547539 (+ 122us) client_negotiation.cc:769] Sending connection context
0923 20:15:45.547634 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 20:15:45.547818 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13085,"mutex_wait_us":45,"thread_start_us":13448,"threads_started":1}
I20260923 20:15:45.548342 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.533000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.5.83.67:33695)
0923 20:15:45.534575 (+ 1575us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:45.534585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:45.535219 (+ 634us) server_negotiation.cc:407] Connection header received
0923 20:15:45.535525 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:45.535536 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:45.535783 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:45.535945 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:45.537034 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.538773 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.545505 (+ 6732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.546914 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.547014 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.547213 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:45.547874 (+ 661us) server_negotiation.cc:299] Negotiation successful
0923 20:15:45.548040 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1371,"thread_start_us":1190,"threads_started":1}
I20260923 20:15:45.549083 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.534506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:33695 (local address 127.0.0.1:47126)
0923 20:15:45.535109 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:45.535128 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:45.535323 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:45.536293 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:45.536300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:45.536331 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:45.536804 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:45.536819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.542705 (+ 5886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.542710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:45.545344 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:45.545358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.547402 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.547409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:45.547657 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.547690 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:15:45.548592 (+ 902us) client_negotiation.cc:240] Negotiation successful
0923 20:15:45.548728 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":39,"thread_start_us":307,"threads_started":1}
I20260923 20:15:45.548839 9689 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6204078841488cac7c7150c99f5b64 (DEFAULT_TABLE table=test-table [id=68fdf85ae0bb4f088020e2e02295523e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:45.550562 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.518438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.5.83.66:46273)
0923 20:15:45.537176 (+ 18738us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:45.537185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:45.537209 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:15:45.537304 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:45.537314 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:45.537518 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:45.537733 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:45.538483 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.540077 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.543100 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.546617 (+ 3517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.546768 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.547034 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:45.549032 (+ 1998us) server_negotiation.cc:299] Negotiation successful
0923 20:15:45.549266 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18556,"thread_start_us":20025,"threads_started":1}
I20260923 20:15:45.550817 9689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6204078841488cac7c7150c99f5b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:45.553205 9954 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6204078841488cac7c7150c99f5b64 (DEFAULT_TABLE table=test-table [id=68fdf85ae0bb4f088020e2e02295523e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:45.554628 9954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6204078841488cac7c7150c99f5b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:45.554463 9822 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6204078841488cac7c7150c99f5b64 (DEFAULT_TABLE table=test-table [id=68fdf85ae0bb4f088020e2e02295523e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:45.557066 9822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6204078841488cac7c7150c99f5b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:45.573256 10037 tablet_bootstrap.cc:492] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b: Bootstrap starting.
I20260923 20:15:45.573999 10038 tablet_bootstrap.cc:492] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142: Bootstrap starting.
I20260923 20:15:45.577957 10038 tablet_bootstrap.cc:654] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:45.579339 10038 log.cc:824] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:45.579952 10037 tablet_bootstrap.cc:654] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:45.581496 10037 log.cc:824] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:45.584749 10039 tablet_bootstrap.cc:492] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c: Bootstrap starting.
I20260923 20:15:45.587296 10038 tablet_bootstrap.cc:492] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142: No bootstrap required, opened a new log
I20260923 20:15:45.587741 10038 ts_tablet_manager.cc:1397] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260923 20:15:45.589838 10039 tablet_bootstrap.cc:654] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:45.591259 10039 log.cc:824] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:45.594926 10037 tablet_bootstrap.cc:492] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b: No bootstrap required, opened a new log
I20260923 20:15:45.595237 10037 ts_tablet_manager.cc:1397] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.009s
I20260923 20:15:45.598999 10039 tablet_bootstrap.cc:492] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c: No bootstrap required, opened a new log
I20260923 20:15:45.599344 10039 ts_tablet_manager.cc:1397] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20260923 20:15:45.599886 10038 raft_consensus.cc:348] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e05e5b675a4a490fb6314d191ab8d03c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46273 } } peers { permanent_uuid: "89dd3c3fd9754d86998371040987392b" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44979 } } peers { permanent_uuid: "989e4acac787450789dfb62005bcb142" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 33695 } }
I20260923 20:15:45.600638 10038 raft_consensus.cc:374] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:45.600788 10038 raft_consensus.cc:729] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 989e4acac787450789dfb62005bcb142, State: Initialized, Role: FOLLOWER
I20260923 20:15:45.601570 10038 consensus_queue.cc:260] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e05e5b675a4a490fb6314d191ab8d03c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46273 } } peers { permanent_uuid: "89dd3c3fd9754d86998371040987392b" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44979 } } peers { permanent_uuid: "989e4acac787450789dfb62005bcb142" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 33695 } }
I20260923 20:15:45.607455 10037 raft_consensus.cc:348] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e05e5b675a4a490fb6314d191ab8d03c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46273 } } peers { permanent_uuid: "89dd3c3fd9754d86998371040987392b" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44979 } } peers { permanent_uuid: "989e4acac787450789dfb62005bcb142" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 33695 } }
I20260923 20:15:45.608033 10037 raft_consensus.cc:374] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:45.608188 10037 raft_consensus.cc:729] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89dd3c3fd9754d86998371040987392b, State: Initialized, Role: FOLLOWER
I20260923 20:15:45.608889 10037 consensus_queue.cc:260] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e05e5b675a4a490fb6314d191ab8d03c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46273 } } peers { permanent_uuid: "89dd3c3fd9754d86998371040987392b" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44979 } } peers { permanent_uuid: "989e4acac787450789dfb62005bcb142" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 33695 } }
I20260923 20:15:45.610042 10038 ts_tablet_manager.cc:1428] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142: Time spent starting tablet: real 0.022s user 0.017s sys 0.003s
I20260923 20:15:45.610626 10019 heartbeater.cc:503] Master 127.5.83.126:38717 was elected leader, sending a full tablet report...
I20260923 20:15:45.615017 10039 raft_consensus.cc:348] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e05e5b675a4a490fb6314d191ab8d03c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46273 } } peers { permanent_uuid: "89dd3c3fd9754d86998371040987392b" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44979 } } peers { permanent_uuid: "989e4acac787450789dfb62005bcb142" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 33695 } }
I20260923 20:15:45.615593 10039 raft_consensus.cc:374] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:45.615742 10039 raft_consensus.cc:729] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05e5b675a4a490fb6314d191ab8d03c, State: Initialized, Role: FOLLOWER
I20260923 20:15:45.616609 10039 consensus_queue.cc:260] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e05e5b675a4a490fb6314d191ab8d03c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46273 } } peers { permanent_uuid: "89dd3c3fd9754d86998371040987392b" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44979 } } peers { permanent_uuid: "989e4acac787450789dfb62005bcb142" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 33695 } }
I20260923 20:15:45.620682 10037 ts_tablet_manager.cc:1428] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b: Time spent starting tablet: real 0.025s user 0.023s sys 0.005s
I20260923 20:15:45.625221 9887 heartbeater.cc:503] Master 127.5.83.126:38717 was elected leader, sending a full tablet report...
I20260923 20:15:45.626091 10039 ts_tablet_manager.cc:1428] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c: Time spent starting tablet: real 0.027s user 0.012s sys 0.012s
W20260923 20:15:45.671088 9888 tablet.cc:2369] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:15:45.679481 10020 tablet.cc:2369] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:15:45.682969 9755 tablet.cc:2369] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:15:45.879854 10044 raft_consensus.cc:482] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:45.880153 10044 raft_consensus.cc:504] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e05e5b675a4a490fb6314d191ab8d03c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46273 } } peers { permanent_uuid: "89dd3c3fd9754d86998371040987392b" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44979 } } peers { permanent_uuid: "989e4acac787450789dfb62005bcb142" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 33695 } }
I20260923 20:15:45.881839 10044 leader_election.cc:288] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e05e5b675a4a490fb6314d191ab8d03c (127.5.83.66:46273), 989e4acac787450789dfb62005bcb142 (127.5.83.67:33695)
I20260923 20:15:45.888170 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.882749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:50369 (local address 127.5.83.66:46273)
0923 20:15:45.882939 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:45.882947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:45.883010 (+ 63us) server_negotiation.cc:407] Connection header received
0923 20:15:45.883260 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:45.883264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:45.883321 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:45.883412 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:45.883905 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.884645 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.886440 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.887232 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.887391 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.887509 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:45.887785 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 20:15:45.887913 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:15:45.888314 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.882706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:58423 (local address 127.5.83.67:33695)
0923 20:15:45.882904 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:45.882912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:45.883314 (+ 402us) server_negotiation.cc:407] Connection header received
0923 20:15:45.883508 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:45.883511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:45.883568 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:45.883665 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:15:45.884179 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.884906 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.886590 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.887365 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.887507 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.887617 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:45.887921 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 20:15:45.888056 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:15:45.888329 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.882203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:46273 (local address 127.5.83.65:50369)
0923 20:15:45.882881 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:45.882905 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:45.883102 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:45.883440 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:45.883444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:45.883476 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:45.883772 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:45.883780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.884773 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.884780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:45.886278 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:45.886289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.887445 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.887451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:45.887626 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.887650 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:15:45.887761 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 20:15:45.887891 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":1,"thread_start_us":312,"threads_started":1}
I20260923 20:15:45.888379 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.882255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:33695 (local address 127.5.83.65:58423)
0923 20:15:45.883204 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:45.883223 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:45.883366 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:45.883686 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:45.883690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:45.883713 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:15:45.884042 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:45.884049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.885046 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.885052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:45.886441 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:45.886453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.887602 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.887608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:45.887768 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.887790 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:15:45.887894 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:15:45.888016 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":665,"threads_started":1}
I20260923 20:15:45.889487 9974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6204078841488cac7c7150c99f5b64" candidate_uuid: "89dd3c3fd9754d86998371040987392b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "989e4acac787450789dfb62005bcb142" is_pre_election: true
I20260923 20:15:45.889482 9842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6204078841488cac7c7150c99f5b64" candidate_uuid: "89dd3c3fd9754d86998371040987392b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e05e5b675a4a490fb6314d191ab8d03c" is_pre_election: true
I20260923 20:15:45.889946 9974 raft_consensus.cc:2454] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd3c3fd9754d86998371040987392b in term 0.
I20260923 20:15:45.889947 9842 raft_consensus.cc:2454] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89dd3c3fd9754d86998371040987392b in term 0.
I20260923 20:15:45.890537 9642 leader_election.cc:302] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89dd3c3fd9754d86998371040987392b, 989e4acac787450789dfb62005bcb142; no voters:
I20260923 20:15:45.890913 10044 raft_consensus.cc:2781] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:15:45.891083 10044 raft_consensus.cc:482] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:15:45.891189 10044 raft_consensus.cc:3037] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:45.893477 10044 raft_consensus.cc:504] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e05e5b675a4a490fb6314d191ab8d03c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46273 } } peers { permanent_uuid: "89dd3c3fd9754d86998371040987392b" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44979 } } peers { permanent_uuid: "989e4acac787450789dfb62005bcb142" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 33695 } }
I20260923 20:15:45.894052 10044 leader_election.cc:288] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [CANDIDATE]: Term 1 election: Requested vote from peers e05e5b675a4a490fb6314d191ab8d03c (127.5.83.66:46273), 989e4acac787450789dfb62005bcb142 (127.5.83.67:33695)
I20260923 20:15:45.894595 9974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6204078841488cac7c7150c99f5b64" candidate_uuid: "89dd3c3fd9754d86998371040987392b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "989e4acac787450789dfb62005bcb142"
I20260923 20:15:45.894595 9842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6204078841488cac7c7150c99f5b64" candidate_uuid: "89dd3c3fd9754d86998371040987392b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e05e5b675a4a490fb6314d191ab8d03c"
I20260923 20:15:45.894821 9974 raft_consensus.cc:3037] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:45.894810 9842 raft_consensus.cc:3037] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:45.898085 9842 raft_consensus.cc:2454] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd3c3fd9754d86998371040987392b in term 1.
I20260923 20:15:45.898079 9974 raft_consensus.cc:2454] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89dd3c3fd9754d86998371040987392b in term 1.
I20260923 20:15:45.898483 9642 leader_election.cc:302] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89dd3c3fd9754d86998371040987392b, 989e4acac787450789dfb62005bcb142; no voters:
I20260923 20:15:45.898772 10044 raft_consensus.cc:2781] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:45.899211 10044 raft_consensus.cc:686] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 1 LEADER]: Becoming Leader. State: Replica: 89dd3c3fd9754d86998371040987392b, State: Running, Role: LEADER
I20260923 20:15:45.899716 10044 consensus_queue.cc:237] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e05e5b675a4a490fb6314d191ab8d03c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46273 } } peers { permanent_uuid: "89dd3c3fd9754d86998371040987392b" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44979 } } peers { permanent_uuid: "989e4acac787450789dfb62005bcb142" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 33695 } }
I20260923 20:15:45.903903 9566 catalog_manager.cc:5495] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b reported cstate change: term changed from 0 to 1, leader changed from <none> to 89dd3c3fd9754d86998371040987392b (127.5.83.65). New cstate: current_term: 1 leader_uuid: "89dd3c3fd9754d86998371040987392b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e05e5b675a4a490fb6314d191ab8d03c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89dd3c3fd9754d86998371040987392b" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "989e4acac787450789dfb62005bcb142" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 33695 } health_report { overall_health: UNKNOWN } } }
I20260923 20:15:45.998865 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:45.993211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.5.83.65:44979)
0923 20:15:45.993390 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:45.993395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:45.993913 (+ 518us) server_negotiation.cc:407] Connection header received
0923 20:15:45.994093 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:45.994096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:45.994184 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:45.994311 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:15:45.994718 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.995425 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.996860 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:45.997282 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:45.997388 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:45.997812 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:15:45.998324 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:15:45.998436 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:45.998583 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 20:15:45.998662 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:15:46.007375 9974 raft_consensus.cc:1261] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142 [term 1 FOLLOWER]: Refusing update from remote peer 89dd3c3fd9754d86998371040987392b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:46.007411 9842 raft_consensus.cc:1261] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c [term 1 FOLLOWER]: Refusing update from remote peer 89dd3c3fd9754d86998371040987392b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:46.008423 10044 consensus_queue.cc:1035] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [LEADER]: Connected to new peer: Peer: permanent_uuid: "989e4acac787450789dfb62005bcb142" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 33695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:46.008785 10048 consensus_queue.cc:1035] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e05e5b675a4a490fb6314d191ab8d03c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 46273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:46.037900 9566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:46.038160 9566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:46.041826 9566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5bad87a8a9e24c898a96270d37883fd0: Sending DeleteTablet for 3 replicas of tablet 4b6204078841488cac7c7150c99f5b64
I20260923 20:15:46.043087 9689 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b6204078841488cac7c7150c99f5b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:46 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:55642
I20260923 20:15:46.043072 9822 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b6204078841488cac7c7150c99f5b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:46 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:34698
I20260923 20:15:46.043327 9954 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b6204078841488cac7c7150c99f5b64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:46 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:47126
I20260923 20:15:46.045485 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 9627
I20260923 20:15:46.075024 10064 tablet_replica.cc:330] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142: stopping tablet replica
I20260923 20:15:46.075610 10064 raft_consensus.cc:2229] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:46.079566 10064 raft_consensus.cc:2258] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:46.080888 10062 tablet_replica.cc:330] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b: stopping tablet replica
I20260923 20:15:46.082504 10064 ts_tablet_manager.cc:1905] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:46.083047 10062 raft_consensus.cc:2229] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:15:46.083887 10063 tablet_replica.cc:330] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c: stopping tablet replica
I20260923 20:15:46.084246 10063 raft_consensus.cc:2229] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:46.084254 10062 raft_consensus.cc:2258] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:46.084817 10063 raft_consensus.cc:2258] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 23 20:15:46 dist-test-slave-vl24 krb5kdc[9522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:46.088647 10063 ts_tablet_manager.cc:1905] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:46.093227 10062 ts_tablet_manager.cc:1905] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:46.096012 10063 ts_tablet_manager.cc:1918] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:15:46.096244 10063 log.cc:1196] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/4b6204078841488cac7c7150c99f5b64
I20260923 20:15:46.096906 10063 ts_tablet_manager.cc:1939] T 4b6204078841488cac7c7150c99f5b64 P e05e5b675a4a490fb6314d191ab8d03c: Deleting consensus metadata
I20260923 20:15:46.098547 9551 catalog_manager.cc:4841] TS e05e5b675a4a490fb6314d191ab8d03c (127.5.83.66:46273): tablet 4b6204078841488cac7c7150c99f5b64 (table test-table [id=68fdf85ae0bb4f088020e2e02295523e]) successfully deleted
I20260923 20:15:46.099447 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:46.046907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.5.83.65:44979)
0923 20:15:46.047114 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:46.047120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:46.071969 (+ 24849us) server_negotiation.cc:407] Connection header received
0923 20:15:46.072498 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:46.072506 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:46.072643 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:46.072720 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:46.075919 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:46.077232 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:46.081201 (+ 3969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:46.082212 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:46.082348 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:46.089795 (+ 7447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:46.089809 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:46.089826 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:46.089879 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:46.094127 (+ 4248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:46.095148 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:46.095157 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:46.095161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:46.095350 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:46.095863 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:46.095868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:46.095870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:46.096353 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:46.096697 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:46.098980 (+ 2283us) server_negotiation.cc:299] Negotiation successful
0923 20:15:46.099187 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:15:46.100896 9749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:46.100858 10062 ts_tablet_manager.cc:1918] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:15:46.101066 10062 log.cc:1196] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/4b6204078841488cac7c7150c99f5b64
I20260923 20:15:46.101563 10064 ts_tablet_manager.cc:1918] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:15:46.101766 10064 log.cc:1196] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/4b6204078841488cac7c7150c99f5b64
I20260923 20:15:46.102429 10064 ts_tablet_manager.cc:1939] T 4b6204078841488cac7c7150c99f5b64 P 989e4acac787450789dfb62005bcb142: Deleting consensus metadata
I20260923 20:15:46.104357 9550 catalog_manager.cc:4841] TS 989e4acac787450789dfb62005bcb142 (127.5.83.67:33695): tablet 4b6204078841488cac7c7150c99f5b64 (table test-table [id=68fdf85ae0bb4f088020e2e02295523e]) successfully deleted
I20260923 20:15:46.202241 10062 ts_tablet_manager.cc:1939] T 4b6204078841488cac7c7150c99f5b64 P 89dd3c3fd9754d86998371040987392b: Deleting consensus metadata
I20260923 20:15:46.202307 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 9627
W20260923 20:15:46.220781 9550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.65:44979 (error 108)
I20260923 20:15:46.221014 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 9760
I20260923 20:15:46.222965 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:46.222111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44979 (local address 127.0.0.1:55672)
0923 20:15:46.222568 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:46.222704 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":196,"threads_started":1}
W20260923 20:15:46.223255 9550 catalog_manager.cc:4584] TS 89dd3c3fd9754d86998371040987392b (127.5.83.65:44979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b6204078841488cac7c7150c99f5b64: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44979: connect: Connection refused (error 111)
Sep 23 20:15:46 dist-test-slave-vl24 krb5kdc[9522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:15:46.235525 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:46.221825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.5.83.66:46273)
0923 20:15:46.222028 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:46.222035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:46.222056 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:15:46.222158 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:46.222163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:46.222259 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:46.222379 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:46.223916 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:46.224800 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:46.226009 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:46.226400 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:46.226494 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:46.230341 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:46.230354 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:46.230367 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:46.230400 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:46.233316 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:46.233904 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:46.233911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:46.233913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:46.234071 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:46.234369 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:46.234372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:46.234374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:46.234708 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:46.234928 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:46.235097 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 20:15:46.235246 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 20:15:46.236212 9882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:46.277179 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:46.276719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44979 (local address 127.0.0.1:55674)
0923 20:15:46.276926 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:46.277033 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 20:15:46.278239 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:46.277909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44979 (local address 127.0.0.1:55676)
0923 20:15:46.278052 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:46.278116 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 20:15:46.326857 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 9760
I20260923 20:15:46.338907 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:46.338305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44979 (local address 127.0.0.1:55686)
0923 20:15:46.338590 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:46.338703 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 20:15:46.340287 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:46.339700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44979 (local address 127.0.0.1:55700)
0923 20:15:46.339986 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:46.340104 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 20:15:46.346948 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 9892
Sep 23 20:15:46 dist-test-slave-vl24 krb5kdc[9522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:15:46.360862 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:46.347711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.5.83.67:33695)
0923 20:15:46.347888 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:46.347893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:46.347925 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:15:46.347979 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:46.347983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:46.348051 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:46.348134 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:46.349444 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:46.350153 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:46.351307 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:46.351694 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:46.351782 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:46.355874 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:46.355887 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:46.355904 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:46.355933 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:46.358595 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:46.359139 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:46.359145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:46.359147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:46.359266 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:46.359602 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:46.359605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:46.359607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:46.359923 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:46.360146 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:46.360355 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 20:15:46.360504 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:15:46.361478 10014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:46.421334 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:46.420857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44979 (local address 127.0.0.1:55708)
0923 20:15:46.421073 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:46.421162 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:15:46.422470 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:46.422082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44979 (local address 127.0.0.1:55718)
0923 20:15:46.422275 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:46.422341 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260923 20:15:46.451864 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 9892
I20260923 20:15:46.468828 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 9535
I20260923 20:15:46.470115 9596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:46.578080 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 9535
2026-09-23T20:15:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4511 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T20:15:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:46Z Disabled control of system clock
I20260923 20:15:46.645964 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:42257
--webserver_interface=127.5.83.126
--webserver_port=0
--builtin_ntp_servers=127.5.83.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 20:15:46.967645 10075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:46.967927 10075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:46.968009 10075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:46.975255 10075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:46.975356 10075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:46.975395 10075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:46.975452 10075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:46.975497 10075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:15:46.985260 10075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:42257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:46.986547 10075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:46.988281 10075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:47.000051 10081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:47.000074 10083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:47.000635 10080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:47.000913 10075 server_base.cc:1034] running on GCE node
I20260923 20:15:47.001628 10075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:47.003042 10075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:47.004287 10075 hybrid_clock.cc:648] HybridClock initialized: now 1790194547004253 us; error 49 us; skew 500 ppm
I20260923 20:15:47.004719 10075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:15:47.013702 10075 webserver.cc:460] Webserver started at http://127.5.83.126:46797/ using document root <none> and password file <none>
I20260923 20:15:47.014451 10075 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:47.014588 10075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:47.014928 10075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:47.017953 10075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "01dd665bb6e6411583f51796aa35727a"
format_stamp: "Formatted at 2026-09-23 20:15:47 on dist-test-slave-vl24"
server_key: "1029ff96c70c30aa3803c3be80f14817"
server_key_iv: "34e46f6abb4296e410c76360a54c545e"
server_key_version: "encryptionkey@0"
I20260923 20:15:47.018765 10075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "01dd665bb6e6411583f51796aa35727a"
format_stamp: "Formatted at 2026-09-23 20:15:47 on dist-test-slave-vl24"
server_key: "1029ff96c70c30aa3803c3be80f14817"
server_key_iv: "34e46f6abb4296e410c76360a54c545e"
server_key_version: "encryptionkey@0"
I20260923 20:15:47.024019 10075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:15:47.027661 10089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:47.029112 10075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:47.029302 10075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "01dd665bb6e6411583f51796aa35727a"
format_stamp: "Formatted at 2026-09-23 20:15:47 on dist-test-slave-vl24"
server_key: "1029ff96c70c30aa3803c3be80f14817"
server_key_iv: "34e46f6abb4296e410c76360a54c545e"
server_key_version: "encryptionkey@0"
I20260923 20:15:47.029542 10075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:47.035277 10075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:47.036051 10075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:47.036350 10075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:47.054510 10075 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:42257
I20260923 20:15:47.054545 10140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:42257 every 8 connection(s)
I20260923 20:15:47.055872 10075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:47.059526 10141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:47.063133 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 10075
I20260923 20:15:47.063364 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:47.063987 5453 external_mini_cluster.cc:1428] Setting key 3a03d5bced261a801229e994aadb623d
I20260923 20:15:47.072006 10141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a: Bootstrap starting.
I20260923 20:15:47.074056 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:47.066040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.5.83.126:42257)
0923 20:15:47.067142 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:47.067157 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:47.067209 (+ 52us) server_negotiation.cc:407] Connection header received
0923 20:15:47.070075 (+ 2866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:47.070105 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:47.070619 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:47.071036 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:47.072036 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:47.072045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:47.072061 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:47.072080 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:47.072173 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:47.072194 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:47.072415 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:47.072580 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 20:15:47.072829 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":338,"threads_started":1}
I20260923 20:15:47.075595 10141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:47.076723 10141 log.cc:824] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:47.079243 10141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a: No bootstrap required, opened a new log
I20260923 20:15:47.086237 10141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01dd665bb6e6411583f51796aa35727a" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42257 } }
I20260923 20:15:47.086632 10141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:47.086745 10141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01dd665bb6e6411583f51796aa35727a, State: Initialized, Role: FOLLOWER
I20260923 20:15:47.087365 10141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01dd665bb6e6411583f51796aa35727a" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42257 } }
I20260923 20:15:47.087623 10141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:47.087764 10141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:47.087946 10141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:47.090226 10141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01dd665bb6e6411583f51796aa35727a" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42257 } }
I20260923 20:15:47.090848 10141 leader_election.cc:302] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01dd665bb6e6411583f51796aa35727a; no voters:
I20260923 20:15:47.091463 10141 leader_election.cc:288] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:47.091579 10146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:47.092921 10146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [term 1 LEADER]: Becoming Leader. State: Replica: 01dd665bb6e6411583f51796aa35727a, State: Running, Role: LEADER
I20260923 20:15:47.093485 10146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01dd665bb6e6411583f51796aa35727a" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42257 } }
I20260923 20:15:47.093775 10141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:47.096338 10148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01dd665bb6e6411583f51796aa35727a. Latest consensus state: current_term: 1 leader_uuid: "01dd665bb6e6411583f51796aa35727a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01dd665bb6e6411583f51796aa35727a" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42257 } } }
I20260923 20:15:47.096776 10148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:47.096431 10147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01dd665bb6e6411583f51796aa35727a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01dd665bb6e6411583f51796aa35727a" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42257 } } }
I20260923 20:15:47.096889 10147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:47.098920 10154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:47.104508 10154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:47.113225 10154 catalog_manager.cc:1347] Generated new cluster ID: fe66acdfcda947bcad6419143c564c9a
I20260923 20:15:47.113368 10154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:47.147923 10154 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:47.148361 10154 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:47.160208 10154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a: Generated new TSK 0
I20260923 20:15:47.161394 10154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:15:47.178273 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42257
--builtin_ntp_servers=127.5.83.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 20:15:47.527932 10165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:47.528335 10165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:47.528539 10165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:47.536813 10165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:47.536969 10165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:47.537171 10165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:15:47.547331 10165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:47.548808 10165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:47.550797 10165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:47.564724 10170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:47.564765 10173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:47.564801 10171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:47.565804 10165 server_base.cc:1034] running on GCE node
I20260923 20:15:47.566391 10165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:47.567301 10165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:47.568511 10165 hybrid_clock.cc:648] HybridClock initialized: now 1790194547568441 us; error 57 us; skew 500 ppm
I20260923 20:15:47.568888 10165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:15:47.571332 10165 webserver.cc:460] Webserver started at http://127.5.83.65:36157/ using document root <none> and password file <none>
I20260923 20:15:47.572113 10165 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:47.572206 10165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:47.572576 10165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:47.575794 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "6b49116a8c10456781f62119599c4569"
format_stamp: "Formatted at 2026-09-23 20:15:47 on dist-test-slave-vl24"
server_key: "f341e17cf51fc2c4afe4cbff2d5e3308"
server_key_iv: "bf803d5353ab26e70442fc308ffac86b"
server_key_version: "encryptionkey@0"
I20260923 20:15:47.576848 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "6b49116a8c10456781f62119599c4569"
format_stamp: "Formatted at 2026-09-23 20:15:47 on dist-test-slave-vl24"
server_key: "f341e17cf51fc2c4afe4cbff2d5e3308"
server_key_iv: "bf803d5353ab26e70442fc308ffac86b"
server_key_version: "encryptionkey@0"
I20260923 20:15:47.582433 10165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:15:47.586057 10179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:47.587445 10165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:15:47.587620 10165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "6b49116a8c10456781f62119599c4569"
format_stamp: "Formatted at 2026-09-23 20:15:47 on dist-test-slave-vl24"
server_key: "f341e17cf51fc2c4afe4cbff2d5e3308"
server_key_iv: "bf803d5353ab26e70442fc308ffac86b"
server_key_version: "encryptionkey@0"
I20260923 20:15:47.587863 10165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:47.593734 10165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:47.594635 10165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:47.594921 10165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:47.595909 10165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:47.597360 10165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:47.597476 10165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:47.597580 10165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:47.597640 10165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:47.630946 10165 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:43857
I20260923 20:15:47.631008 10291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:43857 every 8 connection(s)
I20260923 20:15:47.632315 10165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:15:47.633127 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 10165
I20260923 20:15:47.633366 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:15:47.634195 5453 external_mini_cluster.cc:1428] Setting key d96bcb56df35e8ee85cee1d507741922
I20260923 20:15:47.637666 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42257
--builtin_ntp_servers=127.5.83.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 20:15:47.642759 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:47.635558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:36733 (local address 127.5.83.126:42257)
0923 20:15:47.636052 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:47.636059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:47.637622 (+ 1563us) server_negotiation.cc:407] Connection header received
0923 20:15:47.640398 (+ 2776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:47.640402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:47.640456 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:47.640627 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:47.642093 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:47.642097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:47.642100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:47.642116 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:47.642157 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:47.642169 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:47.642271 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:47.642480 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 20:15:47.642529 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":256,"threads_started":1}
I20260923 20:15:47.644227 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:47.636251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42257 (local address 127.5.83.65:36733)
0923 20:15:47.637446 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:47.637503 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:47.640106 (+ 2603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:47.640840 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:47.640854 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:47.640999 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:47.641721 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:15:47.641727 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:47.641780 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:15:47.641900 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:47.642267 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:47.642279 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:47.642290 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:15:47.642578 (+ 288us) client_negotiation.cc:240] Negotiation successful
0923 20:15:47.642843 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":354,"threads_started":1}
I20260923 20:15:47.646116 10292 heartbeater.cc:347] Connected to a master server at 127.5.83.126:42257
I20260923 20:15:47.646447 10292 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:47.647068 10292 heartbeater.cc:511] Master 127.5.83.126:42257 requested a full tablet report, sending...
I20260923 20:15:47.649286 10106 ts_manager.cc:194] Registered new tserver with Master: 6b49116a8c10456781f62119599c4569 (127.5.83.65:43857)
W20260923 20:15:47.972877 10296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:47.973223 10296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:47.973423 10296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:47.980935 10296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:47.981091 10296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:47.981295 10296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:15:47.990948 10296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:47.992480 10296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:47.994405 10296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:48.008157 10301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:48.008184 10302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:48.008273 10304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:48.009230 10296 server_base.cc:1034] running on GCE node
I20260923 20:15:48.009840 10296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:48.010804 10296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:48.011992 10296 hybrid_clock.cc:648] HybridClock initialized: now 1790194548011941 us; error 54 us; skew 500 ppm
I20260923 20:15:48.012326 10296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:15:48.014564 10296 webserver.cc:460] Webserver started at http://127.5.83.66:33417/ using document root <none> and password file <none>
I20260923 20:15:48.015272 10296 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:48.015384 10296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:48.015722 10296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:48.018617 10296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183"
format_stamp: "Formatted at 2026-09-23 20:15:48 on dist-test-slave-vl24"
server_key: "e3f6b6e21737192e24eda713a195613c"
server_key_iv: "13e3cba255fb688c7fe367fd0cb63797"
server_key_version: "encryptionkey@0"
I20260923 20:15:48.019372 10296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183"
format_stamp: "Formatted at 2026-09-23 20:15:48 on dist-test-slave-vl24"
server_key: "e3f6b6e21737192e24eda713a195613c"
server_key_iv: "13e3cba255fb688c7fe367fd0cb63797"
server_key_version: "encryptionkey@0"
I20260923 20:15:48.024416 10296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:15:48.027865 10310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:48.029296 10296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:48.029464 10296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183"
format_stamp: "Formatted at 2026-09-23 20:15:48 on dist-test-slave-vl24"
server_key: "e3f6b6e21737192e24eda713a195613c"
server_key_iv: "13e3cba255fb688c7fe367fd0cb63797"
server_key_version: "encryptionkey@0"
I20260923 20:15:48.029738 10296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:48.035446 10296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:48.036211 10296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:48.036478 10296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:48.037516 10296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:48.038908 10296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:48.039031 10296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:48.039201 10296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:48.039301 10296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:48.075783 10296 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:37751
I20260923 20:15:48.075832 10422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:37751 every 8 connection(s)
I20260923 20:15:48.077195 10296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:15:48.081620 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 10296
I20260923 20:15:48.081808 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:15:48.082238 5453 external_mini_cluster.cc:1428] Setting key c9dc9cc83d1d33040ec78d398bbf4b16
I20260923 20:15:48.084933 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42257
--builtin_ntp_servers=127.5.83.84:41177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 20:15:48.085698 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.079758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:58291 (local address 127.5.83.126:42257)
0923 20:15:48.079930 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:48.079935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:48.081419 (+ 1484us) server_negotiation.cc:407] Connection header received
0923 20:15:48.083230 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:48.083236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:48.083325 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:48.083436 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:48.084792 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:48.084798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:48.084803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:48.084821 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:48.084861 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:48.084873 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:48.084989 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:48.085274 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 20:15:48.085355 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:15:48.086541 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.080298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42257 (local address 127.5.83.66:58291)
0923 20:15:48.081245 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:48.081290 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:48.082972 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:48.083661 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:48.083669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:48.083761 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:48.084462 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:15:48.084465 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:48.084500 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:15:48.084637 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:48.084982 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:48.084991 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:48.085000 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 20:15:48.085337 (+ 337us) client_negotiation.cc:240] Negotiation successful
0923 20:15:48.085590 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":334,"threads_started":1}
I20260923 20:15:48.089583 10423 heartbeater.cc:347] Connected to a master server at 127.5.83.126:42257
I20260923 20:15:48.089901 10423 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:48.090543 10423 heartbeater.cc:511] Master 127.5.83.126:42257 requested a full tablet report, sending...
I20260923 20:15:48.092036 10106 ts_manager.cc:194] Registered new tserver with Master: 6ebcc06a9bf64b7a9996ef5d9d9d2183 (127.5.83.66:37751)
W20260923 20:15:48.412164 10426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:48.412451 10426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:48.412604 10426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:48.420050 10426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:48.420198 10426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:48.420333 10426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:15:48.430810 10426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:41177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:48.432039 10426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:48.433941 10426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:48.446898 10431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:48.446981 10432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:48.446938 10434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:48.447685 10426 server_base.cc:1034] running on GCE node
I20260923 20:15:48.448222 10426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:48.449160 10426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:48.450361 10426 hybrid_clock.cc:648] HybridClock initialized: now 1790194548450323 us; error 37 us; skew 500 ppm
I20260923 20:15:48.450629 10426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:15:48.452843 10426 webserver.cc:460] Webserver started at http://127.5.83.67:36345/ using document root <none> and password file <none>
I20260923 20:15:48.453482 10426 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:48.453567 10426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:48.453825 10426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:48.456689 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "1e499ba7baac417091de7bf8d0e82c74"
format_stamp: "Formatted at 2026-09-23 20:15:48 on dist-test-slave-vl24"
server_key: "be6ad4827ca7bafd2b08c285d1a6ff99"
server_key_iv: "52c10e0f17db921e8a833e5b72fa9e4e"
server_key_version: "encryptionkey@0"
I20260923 20:15:48.457424 10426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "1e499ba7baac417091de7bf8d0e82c74"
format_stamp: "Formatted at 2026-09-23 20:15:48 on dist-test-slave-vl24"
server_key: "be6ad4827ca7bafd2b08c285d1a6ff99"
server_key_iv: "52c10e0f17db921e8a833e5b72fa9e4e"
server_key_version: "encryptionkey@0"
I20260923 20:15:48.462256 10426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:15:48.465596 10440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:48.466920 10426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:15:48.467096 10426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "1e499ba7baac417091de7bf8d0e82c74"
format_stamp: "Formatted at 2026-09-23 20:15:48 on dist-test-slave-vl24"
server_key: "be6ad4827ca7bafd2b08c285d1a6ff99"
server_key_iv: "52c10e0f17db921e8a833e5b72fa9e4e"
server_key_version: "encryptionkey@0"
I20260923 20:15:48.467312 10426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:48.472860 10426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:48.473738 10426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:48.474000 10426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:48.474886 10426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:48.476289 10426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:48.476408 10426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:48.476580 10426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:48.476698 10426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260923 20:15:48.510386 10426 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:44351
I20260923 20:15:48.510432 10552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:44351 every 8 connection(s)
I20260923 20:15:48.511699 10426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:15:48.516798 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 10426
I20260923 20:15:48.517001 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:15:48.517424 5453 external_mini_cluster.cc:1428] Setting key 9440fea8568d90d70122e8affb8cd5b3
I20260923 20:15:48.520215 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.514239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:44977 (local address 127.5.83.126:42257)
0923 20:15:48.514443 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:48.514449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:48.515889 (+ 1440us) server_negotiation.cc:407] Connection header received
0923 20:15:48.517867 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:48.517873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:48.517954 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:48.518074 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:48.519491 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:48.519496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:48.519500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:48.519518 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:48.519549 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:48.519560 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:48.519654 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:48.519872 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 20:15:48.519919 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:15:48.521231 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.514804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42257 (local address 127.5.83.67:44977)
0923 20:15:48.515729 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:48.515769 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:48.517598 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:48.518324 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:48.518336 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:48.518455 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:48.519173 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:15:48.519175 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:48.519213 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:15:48.519325 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:48.519664 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:48.519675 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:48.519682 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 20:15:48.519954 (+ 272us) client_negotiation.cc:240] Negotiation successful
0923 20:15:48.520252 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":301,"threads_started":1}
I20260923 20:15:48.522554 10553 heartbeater.cc:347] Connected to a master server at 127.5.83.126:42257
I20260923 20:15:48.522841 10553 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:48.523393 10553 heartbeater.cc:511] Master 127.5.83.126:42257 requested a full tablet report, sending...
I20260923 20:15:48.524883 10106 ts_manager.cc:194] Registered new tserver with Master: 1e499ba7baac417091de7bf8d0e82c74 (127.5.83.67:44351)
I20260923 20:15:48.533159 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:15:48.540251 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.537805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46552 (local address 127.5.83.126:42257)
0923 20:15:48.538029 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:48.538037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:48.538479 (+ 442us) server_negotiation.cc:407] Connection header received
0923 20:15:48.538654 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:48.538658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:48.538716 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:48.538810 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:48.539640 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:48.539645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:48.539648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:48.539664 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:48.539695 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:48.539708 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:48.539819 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:48.539856 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 20:15:48.539901 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:15:48.544245 10106 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:15:48.551643 10106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:15:48.580384 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.564151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.5.83.66:37751)
0923 20:15:48.564914 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:48.564927 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:48.574334 (+ 9407us) server_negotiation.cc:407] Connection header received
0923 20:15:48.576704 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:48.576721 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:48.577082 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:48.577251 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:48.579358 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:48.579368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:48.579382 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:48.579412 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:48.579471 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:48.579492 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:48.579678 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:48.579858 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 20:15:48.579965 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":448,"threads_started":1}
I20260923 20:15:48.580477 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.563849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:37751 (local address 127.0.0.1:44910)
0923 20:15:48.574053 (+ 10204us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:48.574135 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:48.576371 (+ 2236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:48.577934 (+ 1563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:48.577944 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:48.578079 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:48.579085 (+ 1006us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:15:48.579088 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:48.579112 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:15:48.579180 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:48.579627 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:48.579640 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:48.579648 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:15:48.579874 (+ 226us) client_negotiation.cc:240] Negotiation successful
0923 20:15:48.579965 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9791,"thread_start_us":13753,"threads_started":1}
I20260923 20:15:48.583518 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.565873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:44351 (local address 127.0.0.1:34866)
0923 20:15:48.577323 (+ 11450us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:48.577352 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:48.577467 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:48.581477 (+ 4010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:48.581483 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:48.581514 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:48.582197 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:15:48.582203 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:48.582229 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:15:48.582294 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:48.582720 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:48.582728 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:48.582735 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 20:15:48.582914 (+ 179us) client_negotiation.cc:240] Negotiation successful
0923 20:15:48.583009 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11302,"thread_start_us":9581,"threads_started":1}
I20260923 20:15:48.583518 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.564697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.5.83.65:43857)
0923 20:15:48.579634 (+ 14937us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:48.579645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:48.579677 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:15:48.579752 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:48.579764 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:48.580008 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:48.580170 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:48.582528 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:48.582543 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:48.582553 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:48.582579 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:48.582628 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:48.582673 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:48.582815 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:48.583026 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 20:15:48.583126 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14732,"thread_start_us":14578,"threads_started":1}
I20260923 20:15:48.583899 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.566093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34866 (local address 127.5.83.67:44351)
0923 20:15:48.580898 (+ 14805us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:48.580906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:48.580939 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:15:48.581011 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:48.581017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:48.581225 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:48.581437 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:48.582456 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:48.582464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:48.582477 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:48.582495 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:48.582564 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:48.582583 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:48.582766 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:48.583432 (+ 666us) server_negotiation.cc:299] Negotiation successful
0923 20:15:48.583520 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14601,"thread_start_us":12629,"threads_started":1}
I20260923 20:15:48.583992 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.564005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43857 (local address 127.0.0.1:37506)
0923 20:15:48.575829 (+ 11824us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:48.575861 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:48.576059 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:48.581235 (+ 5176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:48.581242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:48.581267 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:48.582322 (+ 1055us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:15:48.582327 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:48.582345 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:15:48.582387 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:48.582853 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:48.582857 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:48.582862 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:15:48.583021 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 20:15:48.583079 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11622,"thread_start_us":9406,"threads_started":1}
I20260923 20:15:48.585220 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 13393f1dd6d347aaac83750e88d4be5d (DEFAULT_TABLE table=test-table [id=06423dc695384392924977ec92386e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:48.586768 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13393f1dd6d347aaac83750e88d4be5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:48.587724 10227 tablet_service.cc:1467] Processing CreateTablet for tablet 13393f1dd6d347aaac83750e88d4be5d (DEFAULT_TABLE table=test-table [id=06423dc695384392924977ec92386e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:48.589084 10227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13393f1dd6d347aaac83750e88d4be5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:48.589815 10488 tablet_service.cc:1467] Processing CreateTablet for tablet 13393f1dd6d347aaac83750e88d4be5d (DEFAULT_TABLE table=test-table [id=06423dc695384392924977ec92386e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:15:48.590991 10488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13393f1dd6d347aaac83750e88d4be5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:48.601325 10571 tablet_bootstrap.cc:492] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183: Bootstrap starting.
I20260923 20:15:48.603348 10572 tablet_bootstrap.cc:492] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74: Bootstrap starting.
I20260923 20:15:48.605943 10573 tablet_bootstrap.cc:492] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569: Bootstrap starting.
I20260923 20:15:48.607014 10571 tablet_bootstrap.cc:654] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:48.608011 10572 tablet_bootstrap.cc:654] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:48.608294 10571 log.cc:824] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:48.608676 10573 tablet_bootstrap.cc:654] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:48.609210 10572 log.cc:824] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:48.609901 10573 log.cc:824] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:48.611495 10571 tablet_bootstrap.cc:492] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183: No bootstrap required, opened a new log
I20260923 20:15:48.611750 10571 ts_tablet_manager.cc:1397] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260923 20:15:48.612192 10572 tablet_bootstrap.cc:492] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74: No bootstrap required, opened a new log
I20260923 20:15:48.612455 10572 ts_tablet_manager.cc:1397] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:15:48.613068 10573 tablet_bootstrap.cc:492] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569: No bootstrap required, opened a new log
I20260923 20:15:48.613313 10573 ts_tablet_manager.cc:1397] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260923 20:15:48.618984 10571 raft_consensus.cc:348] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37751 } } peers { permanent_uuid: "6b49116a8c10456781f62119599c4569" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43857 } } peers { permanent_uuid: "1e499ba7baac417091de7bf8d0e82c74" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44351 } }
I20260923 20:15:48.619431 10571 raft_consensus.cc:374] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:48.619545 10571 raft_consensus.cc:729] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ebcc06a9bf64b7a9996ef5d9d9d2183, State: Initialized, Role: FOLLOWER
I20260923 20:15:48.620189 10571 consensus_queue.cc:260] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37751 } } peers { permanent_uuid: "6b49116a8c10456781f62119599c4569" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43857 } } peers { permanent_uuid: "1e499ba7baac417091de7bf8d0e82c74" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44351 } }
I20260923 20:15:48.621345 10423 heartbeater.cc:503] Master 127.5.83.126:42257 was elected leader, sending a full tablet report...
I20260923 20:15:48.622040 10571 ts_tablet_manager.cc:1428] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260923 20:15:48.623039 10572 raft_consensus.cc:348] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37751 } } peers { permanent_uuid: "6b49116a8c10456781f62119599c4569" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43857 } } peers { permanent_uuid: "1e499ba7baac417091de7bf8d0e82c74" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44351 } }
I20260923 20:15:48.623495 10572 raft_consensus.cc:374] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:48.623597 10572 raft_consensus.cc:729] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e499ba7baac417091de7bf8d0e82c74, State: Initialized, Role: FOLLOWER
I20260923 20:15:48.623688 10573 raft_consensus.cc:348] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37751 } } peers { permanent_uuid: "6b49116a8c10456781f62119599c4569" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43857 } } peers { permanent_uuid: "1e499ba7baac417091de7bf8d0e82c74" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44351 } }
I20260923 20:15:48.624079 10573 raft_consensus.cc:374] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:48.624176 10573 raft_consensus.cc:729] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b49116a8c10456781f62119599c4569, State: Initialized, Role: FOLLOWER
I20260923 20:15:48.624231 10572 consensus_queue.cc:260] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37751 } } peers { permanent_uuid: "6b49116a8c10456781f62119599c4569" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43857 } } peers { permanent_uuid: "1e499ba7baac417091de7bf8d0e82c74" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44351 } }
I20260923 20:15:48.624778 10573 consensus_queue.cc:260] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37751 } } peers { permanent_uuid: "6b49116a8c10456781f62119599c4569" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43857 } } peers { permanent_uuid: "1e499ba7baac417091de7bf8d0e82c74" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44351 } }
I20260923 20:15:48.626014 10553 heartbeater.cc:503] Master 127.5.83.126:42257 was elected leader, sending a full tablet report...
I20260923 20:15:48.626412 10572 ts_tablet_manager.cc:1428] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260923 20:15:48.628105 10292 heartbeater.cc:503] Master 127.5.83.126:42257 was elected leader, sending a full tablet report...
I20260923 20:15:48.628250 10579 raft_consensus.cc:482] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:15:48.628504 10579 raft_consensus.cc:504] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37751 } } peers { permanent_uuid: "6b49116a8c10456781f62119599c4569" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43857 } } peers { permanent_uuid: "1e499ba7baac417091de7bf8d0e82c74" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44351 } }
I20260923 20:15:48.629585 10573 ts_tablet_manager.cc:1428] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
W20260923 20:15:48.635046 10293 tablet.cc:2369] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:15:48.637936 10579 leader_election.cc:288] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ebcc06a9bf64b7a9996ef5d9d9d2183 (127.5.83.66:37751), 1e499ba7baac417091de7bf8d0e82c74 (127.5.83.67:44351)
I20260923 20:15:48.644181 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.632217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:51061 (local address 127.5.83.66:37751)
0923 20:15:48.632384 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:48.632390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:48.641905 (+ 9515us) server_negotiation.cc:407] Connection header received
0923 20:15:48.642180 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:48.642189 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:48.642258 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:48.642356 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:48.643576 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:48.643582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:48.643588 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:48.643606 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:48.643644 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:48.643658 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:48.643777 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:48.643871 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 20:15:48.643925 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:15:48.644347 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.632265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:60005 (local address 127.5.83.67:44351)
0923 20:15:48.632405 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:48.632413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:48.642251 (+ 9838us) server_negotiation.cc:407] Connection header received
0923 20:15:48.642371 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:48.642376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:48.642432 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:48.642543 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:48.643655 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:48.643659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:48.643662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:48.643678 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:48.643730 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:48.643747 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:48.643869 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:48.644055 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 20:15:48.644116 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 20:15:48.644436 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.631726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:37751 (local address 127.5.83.65:51061)
0923 20:15:48.641800 (+ 10074us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:48.641824 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:48.642005 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:48.642425 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:48.642430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:48.642453 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:48.643344 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:15:48.643350 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:48.643369 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:15:48.643428 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:48.643760 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:48.643767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:48.643775 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:15:48.643920 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 20:15:48.643975 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9930,"thread_start_us":10141,"threads_started":1}
I20260923 20:15:48.645187 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.631761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:44351 (local address 127.5.83.65:60005)
0923 20:15:48.642141 (+ 10380us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:48.642158 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:48.642260 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:48.642561 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:48.642566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:48.642593 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:48.643466 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:15:48.643472 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:48.643486 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:15:48.643524 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:48.643921 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:48.643926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:48.643931 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:15:48.644689 (+ 758us) client_negotiation.cc:240] Negotiation successful
0923 20:15:48.644743 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10256,"thread_start_us":9707,"threads_started":1}
I20260923 20:15:48.645520 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13393f1dd6d347aaac83750e88d4be5d" candidate_uuid: "6b49116a8c10456781f62119599c4569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183" is_pre_election: true
I20260923 20:15:48.646068 10378 raft_consensus.cc:2454] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b49116a8c10456781f62119599c4569 in term 0.
I20260923 20:15:48.646261 10508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13393f1dd6d347aaac83750e88d4be5d" candidate_uuid: "6b49116a8c10456781f62119599c4569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e499ba7baac417091de7bf8d0e82c74" is_pre_election: true
I20260923 20:15:48.646728 10508 raft_consensus.cc:2454] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b49116a8c10456781f62119599c4569 in term 0.
I20260923 20:15:48.646759 10181 leader_election.cc:302] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b49116a8c10456781f62119599c4569, 6ebcc06a9bf64b7a9996ef5d9d9d2183; no voters:
I20260923 20:15:48.647164 10579 raft_consensus.cc:2781] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:15:48.647327 10579 raft_consensus.cc:482] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:15:48.647447 10579 raft_consensus.cc:3037] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:48.649765 10579 raft_consensus.cc:504] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37751 } } peers { permanent_uuid: "6b49116a8c10456781f62119599c4569" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43857 } } peers { permanent_uuid: "1e499ba7baac417091de7bf8d0e82c74" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44351 } }
I20260923 20:15:48.650418 10579 leader_election.cc:288] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [CANDIDATE]: Term 1 election: Requested vote from peers 6ebcc06a9bf64b7a9996ef5d9d9d2183 (127.5.83.66:37751), 1e499ba7baac417091de7bf8d0e82c74 (127.5.83.67:44351)
I20260923 20:15:48.650786 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13393f1dd6d347aaac83750e88d4be5d" candidate_uuid: "6b49116a8c10456781f62119599c4569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183"
I20260923 20:15:48.650980 10378 raft_consensus.cc:3037] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:48.650944 10508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13393f1dd6d347aaac83750e88d4be5d" candidate_uuid: "6b49116a8c10456781f62119599c4569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e499ba7baac417091de7bf8d0e82c74"
I20260923 20:15:48.651151 10508 raft_consensus.cc:3037] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:48.653707 10508 raft_consensus.cc:2454] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b49116a8c10456781f62119599c4569 in term 1.
I20260923 20:15:48.653736 10378 raft_consensus.cc:2454] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b49116a8c10456781f62119599c4569 in term 1.
I20260923 20:15:48.654135 10183 leader_election.cc:302] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e499ba7baac417091de7bf8d0e82c74, 6b49116a8c10456781f62119599c4569; no voters:
I20260923 20:15:48.654428 10579 raft_consensus.cc:2781] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:48.654860 10579 raft_consensus.cc:686] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 1 LEADER]: Becoming Leader. State: Replica: 6b49116a8c10456781f62119599c4569, State: Running, Role: LEADER
I20260923 20:15:48.655392 10579 consensus_queue.cc:237] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37751 } } peers { permanent_uuid: "6b49116a8c10456781f62119599c4569" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43857 } } peers { permanent_uuid: "1e499ba7baac417091de7bf8d0e82c74" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44351 } }
I20260923 20:15:48.659673 10105 catalog_manager.cc:5495] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b49116a8c10456781f62119599c4569 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "6b49116a8c10456781f62119599c4569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b49116a8c10456781f62119599c4569" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e499ba7baac417091de7bf8d0e82c74" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44351 } health_report { overall_health: UNKNOWN } } }
I20260923 20:15:48.682037 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.680043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.5.83.65:43857)
0923 20:15:48.680239 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:48.680243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:48.680262 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:15:48.680449 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:48.680453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:48.680579 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:48.680673 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:48.681458 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:48.681462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:48.681465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:48.681480 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:48.681511 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:48.681523 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:48.681615 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:48.681691 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 20:15:48.681734 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:15:48.690097 10508 raft_consensus.cc:1261] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74 [term 1 FOLLOWER]: Refusing update from remote peer 6b49116a8c10456781f62119599c4569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:48.690105 10378 raft_consensus.cc:1261] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183 [term 1 FOLLOWER]: Refusing update from remote peer 6b49116a8c10456781f62119599c4569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:15:48.691078 10582 consensus_queue.cc:1035] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e499ba7baac417091de7bf8d0e82c74" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:48.691421 10579 consensus_queue.cc:1035] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebcc06a9bf64b7a9996ef5d9d9d2183" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:15:48.702906 10585 mvcc.cc:205] Tried to move back new op lower bound from 7332636871424618496 to 7332636871296069632. Current Snapshot: MvccSnapshot[applied={T|T < 7332636871424618496}]
I20260923 20:15:48.708619 10586 mvcc.cc:205] Tried to move back new op lower bound from 7332636871424618496 to 7332636871296069632. Current Snapshot: MvccSnapshot[applied={T|T < 7332636871424618496}]
I20260923 20:15:48.716957 10106 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:48.717207 10106 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:15:48.720651 10106 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01dd665bb6e6411583f51796aa35727a: Sending DeleteTablet for 3 replicas of tablet 13393f1dd6d347aaac83750e88d4be5d
I20260923 20:15:48.721962 10358 tablet_service.cc:1514] Processing DeleteTablet for tablet 13393f1dd6d347aaac83750e88d4be5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:48 UTC) from {username='slave'} at 127.0.0.1:44910
I20260923 20:15:48.722713 10227 tablet_service.cc:1514] Processing DeleteTablet for tablet 13393f1dd6d347aaac83750e88d4be5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:48 UTC) from {username='slave'} at 127.0.0.1:37506
I20260923 20:15:48.723356 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 10165
I20260923 20:15:48.723412 10488 tablet_service.cc:1514] Processing DeleteTablet for tablet 13393f1dd6d347aaac83750e88d4be5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:15:48 UTC) from {username='slave'} at 127.0.0.1:34866
I20260923 20:15:48.729138 10596 tablet_replica.cc:330] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569: stopping tablet replica
I20260923 20:15:48.730002 10596 raft_consensus.cc:2229] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:15:48.730825 10596 raft_consensus.cc:2258] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:48.732366 10597 tablet_replica.cc:330] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74: stopping tablet replica
I20260923 20:15:48.732878 10597 raft_consensus.cc:2229] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:48.733417 10597 raft_consensus.cc:2258] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:48.734632 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.724864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.5.83.65:43857)
0923 20:15:48.725049 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:48.725057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:48.731697 (+ 6640us) server_negotiation.cc:407] Connection header received
0923 20:15:48.731917 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:48.731924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:48.732003 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:48.732121 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:48.733535 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:48.733541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:48.733547 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:48.733576 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:48.733630 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:48.733648 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:48.733747 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:48.734274 (+ 527us) server_negotiation.cc:299] Negotiation successful
0923 20:15:48.734339 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:15:48.735875 10287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:48.736064 10597 ts_tablet_manager.cc:1905] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:48.736901 10595 tablet_replica.cc:330] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183: stopping tablet replica
I20260923 20:15:48.737324 10595 raft_consensus.cc:2229] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:15:48.737944 10595 raft_consensus.cc:2258] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:15:48.740396 10595 ts_tablet_manager.cc:1905] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:48.743657 10597 ts_tablet_manager.cc:1918] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:15:48.743839 10597 log.cc:1196] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/13393f1dd6d347aaac83750e88d4be5d
I20260923 20:15:48.744334 10597 ts_tablet_manager.cc:1939] T 13393f1dd6d347aaac83750e88d4be5d P 1e499ba7baac417091de7bf8d0e82c74: Deleting consensus metadata
I20260923 20:15:48.745926 10093 catalog_manager.cc:4841] TS 1e499ba7baac417091de7bf8d0e82c74 (127.5.83.67:44351): tablet 13393f1dd6d347aaac83750e88d4be5d (table test-table [id=06423dc695384392924977ec92386e01]) successfully deleted
I20260923 20:15:48.747594 10595 ts_tablet_manager.cc:1918] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:15:48.747759 10595 log.cc:1196] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/13393f1dd6d347aaac83750e88d4be5d
I20260923 20:15:48.748297 10595 ts_tablet_manager.cc:1939] T 13393f1dd6d347aaac83750e88d4be5d P 6ebcc06a9bf64b7a9996ef5d9d9d2183: Deleting consensus metadata
I20260923 20:15:48.750171 10091 catalog_manager.cc:4841] TS 6ebcc06a9bf64b7a9996ef5d9d9d2183 (127.5.83.66:37751): tablet 13393f1dd6d347aaac83750e88d4be5d (table test-table [id=06423dc695384392924977ec92386e01]) successfully deleted
I20260923 20:15:48.736271 10596 ts_tablet_manager.cc:1905] T 13393f1dd6d347aaac83750e88d4be5d P 6b49116a8c10456781f62119599c4569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:15:48.831883 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 10165
I20260923 20:15:48.850373 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 10296
W20260923 20:15:48.850385 10090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.65:43857 (error 108)
I20260923 20:15:48.852255 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.851820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43857 (local address 127.0.0.1:37528)
0923 20:15:48.851987 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:48.852122 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 20:15:48.852466 10090 catalog_manager.cc:4584] TS 6b49116a8c10456781f62119599c4569 (127.5.83.65:43857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13393f1dd6d347aaac83750e88d4be5d: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43857: connect: Connection refused (error 111)
I20260923 20:15:48.853462 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.851141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.5.83.66:37751)
0923 20:15:48.851324 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:48.851330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:48.851420 (+ 90us) server_negotiation.cc:407] Connection header received
0923 20:15:48.851637 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:48.851643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:48.851759 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:48.851870 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:48.852872 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:48.852877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:48.852880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:48.852895 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:48.852925 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:48.852936 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:48.853058 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:48.853141 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 20:15:48.853184 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:15:48.854048 10418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:48.906162 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.905579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43857 (local address 127.0.0.1:37538)
0923 20:15:48.905827 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:48.905952 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 20:15:48.907239 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.906835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43857 (local address 127.0.0.1:37544)
0923 20:15:48.906997 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:48.907102 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 20:15:48.942983 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 10296
I20260923 20:15:48.959472 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 10426
I20260923 20:15:48.962288 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.960356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.5.83.67:44351)
0923 20:15:48.960586 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:48.960591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:48.960609 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:15:48.960664 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:48.960668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:48.960763 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:48.960846 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:48.961662 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:48.961666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:48.961670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:48.961685 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:48.961717 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:48.961729 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:48.961830 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:48.961900 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 20:15:48.961944 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:15:48.962932 10548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:48.967819 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.967316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43857 (local address 127.0.0.1:37546)
0923 20:15:48.967514 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:48.967642 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 20:15:48.968786 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:48.968387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43857 (local address 127.0.0.1:37548)
0923 20:15:48.968560 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:48.968647 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 20:15:49.049597 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:49.049031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43857 (local address 127.0.0.1:37556)
0923 20:15:49.049275 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:49.049396 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 20:15:49.051714 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:49.050275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43857 (local address 127.0.0.1:37558)
0923 20:15:49.051389 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:49.051489 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260923 20:15:49.052459 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 10426
I20260923 20:15:49.074441 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 10075
I20260923 20:15:49.075702 10136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:49.187443 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 10075
2026-09-23T20:15:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2609 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 20:15:49.231149 5453 mini_oidc.cc:152] Starting JWKS server
I20260923 20:15:49.234154 5453 webserver.cc:460] Webserver started at https://127.0.0.1:39905/ using document root <none> and password file <none>
I20260923 20:15:49.234505 5453 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 20:15:49.235167 5453 webserver.cc:460] Webserver started at http://127.0.0.1:42161/ using document root <none> and password file <none>
2026-09-23T20:15:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:49Z Disabled control of system clock
I20260923 20:15:49.263859 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:37687
--webserver_interface=127.5.83.126
--webserver_port=0
--builtin_ntp_servers=127.5.83.84:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:37687
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39905/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/testchainca.cert with env {}
W20260923 20:15:49.594890 10609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:49.595263 10609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:49.595429 10609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:49.602680 10609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:49.602841 10609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:49.602938 10609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:49.603048 10609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:49.603137 10609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:15:49.603227 10609 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:15:49.603303 10609 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39905/jwks.json
I20260923 20:15:49.613341 10609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:37687
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39905/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:49.614719 10609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:49.616041 10609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:49.627815 10617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:49.627944 10614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:49.627868 10615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:49.629174 10609 server_base.cc:1034] running on GCE node
I20260923 20:15:49.630014 10609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:49.631467 10609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:49.632711 10609 hybrid_clock.cc:648] HybridClock initialized: now 1790194549632649 us; error 57 us; skew 500 ppm
I20260923 20:15:49.633109 10609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:15:49.635329 10609 webserver.cc:460] Webserver started at http://127.5.83.126:45839/ using document root <none> and password file <none>
I20260923 20:15:49.636055 10609 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:49.636196 10609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:49.636515 10609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:49.639557 10609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "00edd55f43f64689b14977a29f7c26af"
format_stamp: "Formatted at 2026-09-23 20:15:49 on dist-test-slave-vl24"
server_key: "4e15c0e22ad9b1f05cfc4be5045ac9e1"
server_key_iv: "3609d0a87ce1df3f56f34bb4d39bbf06"
server_key_version: "encryptionkey@0"
I20260923 20:15:49.640385 10609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "00edd55f43f64689b14977a29f7c26af"
format_stamp: "Formatted at 2026-09-23 20:15:49 on dist-test-slave-vl24"
server_key: "4e15c0e22ad9b1f05cfc4be5045ac9e1"
server_key_iv: "3609d0a87ce1df3f56f34bb4d39bbf06"
server_key_version: "encryptionkey@0"
I20260923 20:15:49.645651 10609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:15:49.648989 10623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:49.650300 10609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:15:49.650489 10609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "00edd55f43f64689b14977a29f7c26af"
format_stamp: "Formatted at 2026-09-23 20:15:49 on dist-test-slave-vl24"
server_key: "4e15c0e22ad9b1f05cfc4be5045ac9e1"
server_key_iv: "3609d0a87ce1df3f56f34bb4d39bbf06"
server_key_version: "encryptionkey@0"
I20260923 20:15:49.650708 10609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:49.692672 10609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:49.693570 10609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:49.693866 10609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:49.712208 10609 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:37687
I20260923 20:15:49.712231 10674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:37687 every 8 connection(s)
I20260923 20:15:49.713649 10609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:49.717509 10675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:49.724243 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 10609
I20260923 20:15:49.724459 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:49.725262 5453 external_mini_cluster.cc:1428] Setting key 643feac800f39bda76d661cf2e70e3cb
I20260923 20:15:49.728837 10675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af: Bootstrap starting.
I20260923 20:15:49.732131 10675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:49.733251 10675 log.cc:824] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:49.736068 10675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af: No bootstrap required, opened a new log
I20260923 20:15:49.739019 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:49.727271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.5.83.126:37687)
0923 20:15:49.728219 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:49.728230 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:49.728271 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:15:49.730069 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:49.730094 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:49.730562 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:49.730956 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:49.731521 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:49.733043 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:49.734428 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:49.735114 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:49.735315 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:49.736621 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:49.736630 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:49.736647 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:49.737208 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:49.737283 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:49.737303 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:49.737434 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:49.737692 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 20:15:49.738020 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":324,"threads_started":1}
I20260923 20:15:49.742995 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:15:49.744298 10675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00edd55f43f64689b14977a29f7c26af" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37687 } }
I20260923 20:15:49.744745 10675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:49.744853 10675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00edd55f43f64689b14977a29f7c26af, State: Initialized, Role: FOLLOWER
I20260923 20:15:49.745525 10675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00edd55f43f64689b14977a29f7c26af" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37687 } }
I20260923 20:15:49.745792 10675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:49.745949 10675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:49.746150 10675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:49.748499 10675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00edd55f43f64689b14977a29f7c26af" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37687 } }
I20260923 20:15:49.749171 10675 leader_election.cc:302] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00edd55f43f64689b14977a29f7c26af; no voters:
I20260923 20:15:49.749761 10675 leader_election.cc:288] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:49.749893 10680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:49.751171 10680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [term 1 LEADER]: Becoming Leader. State: Replica: 00edd55f43f64689b14977a29f7c26af, State: Running, Role: LEADER
I20260923 20:15:49.751677 10680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00edd55f43f64689b14977a29f7c26af" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37687 } }
I20260923 20:15:49.752082 10675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:49.756780 10682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00edd55f43f64689b14977a29f7c26af. Latest consensus state: current_term: 1 leader_uuid: "00edd55f43f64689b14977a29f7c26af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00edd55f43f64689b14977a29f7c26af" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37687 } } }
I20260923 20:15:49.757237 10682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:49.757702 10681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00edd55f43f64689b14977a29f7c26af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00edd55f43f64689b14977a29f7c26af" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37687 } } }
I20260923 20:15:49.757998 10681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:49.758311 10692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:49.764213 10692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:49.772328 10692 catalog_manager.cc:1347] Generated new cluster ID: 1e6b2406d8bc494eb833516aafc75ea3
I20260923 20:15:49.772459 10692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:49.797755 10692 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:49.799084 10692 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:49.810596 10692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00edd55f43f64689b14977a29f7c26af: Generated new TSK 0
I20260923 20:15:49.811405 10692 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 20:15:49.855042 10705 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 20:15:49.855244 10705 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 20:15:49.856453 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:49.835570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.5.83.126:37687)
0923 20:15:49.835772 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:49.835797 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:49.836205 (+ 408us) server_negotiation.cc:407] Connection header received
0923 20:15:49.836621 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:49.836625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:49.836725 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:49.836874 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:15:49.837396 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:49.838126 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:49.839510 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:49.839954 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:49.840089 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:49.840649 (+ 560us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:15:49.855762 (+ 15113us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 20:15:49.856160 (+ 398us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 20:15:49.856599 10678 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 20:15:49.857987 10704 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:15:49.835446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:37687 (local address 127.0.0.1:38572)
0923 20:15:49.836096 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:49.836117 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:49.836326 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:49.836908 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:49.836912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:49.836927 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:15:49.837244 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:49.837252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:49.838272 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:49.838278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:49.839361 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:49.839372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:49.840138 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:49.840146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:49.840301 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:49.840358 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:15:49.856065 (+ 15707us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 20:15:49.856394 (+ 329us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.83.126:37687: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":304,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.83.126:37687: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 20:15:49.859699 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 10609
I20260923 20:15:49.860839 10670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:49.956192 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 10609
2026-09-23T20:15:49Z chronyd exiting
I20260923 20:15:50.007303 5453 test_util.cc:175] -----------------------------------------------
I20260923 20:15:50.007462 5453 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790194506409421-5453-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (781 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 20:15:50.011358 5453 mini_oidc.cc:152] Starting JWKS server
I20260923 20:15:50.013298 5453 webserver.cc:460] Webserver started at https://127.0.0.1:38579/ using document root <none> and password file <none>
I20260923 20:15:50.013551 5453 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 20:15:50.014215 5453 webserver.cc:460] Webserver started at http://127.0.0.1:44467/ using document root <none> and password file <none>
2026-09-23T20:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:50Z Disabled control of system clock
I20260923 20:15:50.042452 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:43733
--webserver_interface=127.5.83.126
--webserver_port=0
--builtin_ntp_servers=127.5.83.84:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:43733
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38579/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/testchainca.cert with env {}
W20260923 20:15:50.378386 10712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:50.378722 10712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:50.378875 10712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:50.386163 10712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:50.386343 10712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:50.386435 10712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:50.386536 10712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:50.386611 10712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:15:50.386700 10712 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:15:50.386772 10712 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38579/jwks.json
I20260923 20:15:50.397181 10712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:43733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:43733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38579/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:50.398566 10712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:50.399869 10712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:50.411746 10720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:50.411815 10718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:50.411782 10717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:50.412726 10712 server_base.cc:1034] running on GCE node
I20260923 20:15:50.413552 10712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:50.415032 10712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:50.416262 10712 hybrid_clock.cc:648] HybridClock initialized: now 1790194550416209 us; error 52 us; skew 500 ppm
I20260923 20:15:50.416677 10712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:15:50.418911 10712 webserver.cc:460] Webserver started at http://127.5.83.126:35907/ using document root <none> and password file <none>
I20260923 20:15:50.419610 10712 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:50.419744 10712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:50.420071 10712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:50.422991 10712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "4016eb9a76e14ccdb4196e45bac27124"
format_stamp: "Formatted at 2026-09-23 20:15:50 on dist-test-slave-vl24"
server_key: "fe2418ad79e531f9ce954339cc4247e9"
server_key_iv: "fba964dab02200c4de28acdc34cd0a02"
server_key_version: "encryptionkey@0"
I20260923 20:15:50.423789 10712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "4016eb9a76e14ccdb4196e45bac27124"
format_stamp: "Formatted at 2026-09-23 20:15:50 on dist-test-slave-vl24"
server_key: "fe2418ad79e531f9ce954339cc4247e9"
server_key_iv: "fba964dab02200c4de28acdc34cd0a02"
server_key_version: "encryptionkey@0"
I20260923 20:15:50.429507 10712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:15:50.433975 10726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:50.435420 10712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260923 20:15:50.435570 10712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "4016eb9a76e14ccdb4196e45bac27124"
format_stamp: "Formatted at 2026-09-23 20:15:50 on dist-test-slave-vl24"
server_key: "fe2418ad79e531f9ce954339cc4247e9"
server_key_iv: "fba964dab02200c4de28acdc34cd0a02"
server_key_version: "encryptionkey@0"
I20260923 20:15:50.435760 10712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:50.456252 10712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:50.457178 10712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:50.457432 10712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:50.474982 10712 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:43733
I20260923 20:15:50.475026 10777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:43733 every 8 connection(s)
I20260923 20:15:50.476326 10712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:50.480219 10778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:50.482602 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 10712
I20260923 20:15:50.482821 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:50.483590 5453 external_mini_cluster.cc:1428] Setting key d40e328753cf1bd3e4bf6913e6686dc3
I20260923 20:15:50.494027 10778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124: Bootstrap starting.
I20260923 20:15:50.497948 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:50.485565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.5.83.126:43733)
0923 20:15:50.486633 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:50.486644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:50.486685 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:15:50.488300 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:50.488322 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:50.488849 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:50.489162 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:50.489876 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:50.491235 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:50.493392 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:50.494055 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:50.494266 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:50.495625 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:50.495633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:50.495645 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:50.496203 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:50.496282 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:50.496302 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:50.496387 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:50.496716 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 20:15:50.497037 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":366,"threads_started":1}
I20260923 20:15:50.498250 10778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:50.499460 10778 log.cc:824] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:50.502099 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:15:50.502604 10778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124: No bootstrap required, opened a new log
I20260923 20:15:50.509923 10778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4016eb9a76e14ccdb4196e45bac27124" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43733 } }
I20260923 20:15:50.510303 10778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:50.510403 10778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4016eb9a76e14ccdb4196e45bac27124, State: Initialized, Role: FOLLOWER
I20260923 20:15:50.511032 10778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4016eb9a76e14ccdb4196e45bac27124" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43733 } }
I20260923 20:15:50.511294 10778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:50.511437 10778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:50.511613 10778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:50.514056 10778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4016eb9a76e14ccdb4196e45bac27124" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43733 } }
I20260923 20:15:50.514592 10778 leader_election.cc:302] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4016eb9a76e14ccdb4196e45bac27124; no voters:
I20260923 20:15:50.515307 10778 leader_election.cc:288] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:50.515470 10783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:50.516824 10783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [term 1 LEADER]: Becoming Leader. State: Replica: 4016eb9a76e14ccdb4196e45bac27124, State: Running, Role: LEADER
I20260923 20:15:50.517414 10778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:50.517333 10783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4016eb9a76e14ccdb4196e45bac27124" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43733 } }
I20260923 20:15:50.520900 10785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4016eb9a76e14ccdb4196e45bac27124. Latest consensus state: current_term: 1 leader_uuid: "4016eb9a76e14ccdb4196e45bac27124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4016eb9a76e14ccdb4196e45bac27124" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43733 } } }
I20260923 20:15:50.521407 10785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:50.521044 10784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4016eb9a76e14ccdb4196e45bac27124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4016eb9a76e14ccdb4196e45bac27124" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43733 } } }
I20260923 20:15:50.521947 10784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:50.523164 10792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:50.527604 10792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:50.536456 10792 catalog_manager.cc:1347] Generated new cluster ID: 5fc02cc8fff84b53a52fd32da2fb99a3
I20260923 20:15:50.536638 10792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:50.548558 10792 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:50.549907 10792 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:50.557039 10792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4016eb9a76e14ccdb4196e45bac27124: Generated new TSK 0
I20260923 20:15:50.558064 10792 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 20:15:50.598615 10808 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 20:15:50.598766 10808 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 20:15:50.600045 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:50.579739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.5.83.126:43733)
0923 20:15:50.579927 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:50.579934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:50.580297 (+ 363us) server_negotiation.cc:407] Connection header received
0923 20:15:50.580574 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:50.580582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:50.580677 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:50.580840 (+ 163us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:15:50.581323 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:50.582488 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:50.584013 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:50.584746 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:50.584883 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:50.585376 (+ 493us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:15:50.599311 (+ 13935us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38579/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 20:15:50.599716 (+ 405us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38579/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 20:15:50.600127 10807 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:15:50.579583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43733 (local address 127.0.0.1:36230)
0923 20:15:50.580191 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:50.580210 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:50.580340 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:50.580858 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:50.580862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:50.580875 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:15:50.581154 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:50.581163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:50.582654 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:50.582658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:50.583872 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:50.583885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:50.584916 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:50.584921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:50.585073 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:50.585107 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:15:50.599582 (+ 14475us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38579/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 20:15:50.599785 (+ 203us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.83.126:43733: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38579/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":334,"threads_started":1}
W20260923 20:15:50.600191 10781 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38579/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260923 20:15:50.601473 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 10712
I20260923 20:15:50.602783 10773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:50.697407 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 10712
2026-09-23T20:15:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (742 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 20:15:50.754050 5453 mini_oidc.cc:152] Starting JWKS server
I20260923 20:15:50.756079 5453 webserver.cc:460] Webserver started at https://127.0.0.1:36575/ using document root <none> and password file <none>
I20260923 20:15:50.756331 5453 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 20:15:50.757148 5453 webserver.cc:460] Webserver started at http://127.0.0.1:40605/ using document root <none> and password file <none>
2026-09-23T20:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:50Z Disabled control of system clock
I20260923 20:15:50.787436 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:40415
--webserver_interface=127.5.83.126
--webserver_port=0
--builtin_ntp_servers=127.5.83.84:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:40415
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36575/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 20:15:51.111222 10815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:51.111519 10815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:51.111722 10815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:51.118943 10815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:51.119045 10815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:51.119134 10815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:51.119241 10815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:51.119318 10815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:15:51.119406 10815 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:15:51.119464 10815 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36575/jwks.json
I20260923 20:15:51.128707 10815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:40415
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:40415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36575/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:51.130172 10815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:51.131990 10815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:51.143869 10820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:51.143780 10821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:51.143836 10823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:51.144848 10815 server_base.cc:1034] running on GCE node
I20260923 20:15:51.145650 10815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:51.147027 10815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:51.148232 10815 hybrid_clock.cc:648] HybridClock initialized: now 1790194551148189 us; error 39 us; skew 500 ppm
I20260923 20:15:51.148610 10815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:15:51.150923 10815 webserver.cc:460] Webserver started at http://127.5.83.126:46473/ using document root <none> and password file <none>
I20260923 20:15:51.151559 10815 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:51.151644 10815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:51.151901 10815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:51.154815 10815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "5eb32afe32e64f1789c980b0c38967cd"
format_stamp: "Formatted at 2026-09-23 20:15:51 on dist-test-slave-vl24"
server_key: "5b9732355fc42e18dcaafd2810917d6f"
server_key_iv: "0349ee4ec37b43846555639ff19efcea"
server_key_version: "encryptionkey@0"
I20260923 20:15:51.155573 10815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "5eb32afe32e64f1789c980b0c38967cd"
format_stamp: "Formatted at 2026-09-23 20:15:51 on dist-test-slave-vl24"
server_key: "5b9732355fc42e18dcaafd2810917d6f"
server_key_iv: "0349ee4ec37b43846555639ff19efcea"
server_key_version: "encryptionkey@0"
I20260923 20:15:51.160718 10815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:15:51.164520 10829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:51.166402 10815 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:15:51.166580 10815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "5eb32afe32e64f1789c980b0c38967cd"
format_stamp: "Formatted at 2026-09-23 20:15:51 on dist-test-slave-vl24"
server_key: "5b9732355fc42e18dcaafd2810917d6f"
server_key_iv: "0349ee4ec37b43846555639ff19efcea"
server_key_version: "encryptionkey@0"
I20260923 20:15:51.166827 10815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:51.181046 10815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:51.181891 10815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:51.182186 10815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:51.199822 10815 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:40415
I20260923 20:15:51.199846 10880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:40415 every 8 connection(s)
I20260923 20:15:51.201202 10815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:51.204962 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:51.207800 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 10815
I20260923 20:15:51.208017 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:51.208863 5453 external_mini_cluster.cc:1428] Setting key 71bd181f75ee0432f680d7023abb5745
I20260923 20:15:51.217147 10881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd: Bootstrap starting.
I20260923 20:15:51.220088 10881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:51.221237 10881 log.cc:824] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:51.222404 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:51.210827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.5.83.126:40415)
0923 20:15:51.211803 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:51.211814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:51.211852 (+ 38us) server_negotiation.cc:407] Connection header received
0923 20:15:51.213520 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:51.213541 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:51.214003 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:51.214411 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:51.214963 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:51.216361 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:51.217766 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:51.218418 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:51.218618 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:51.219749 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:51.219756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:51.219769 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:51.220364 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:51.220460 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:51.220483 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:51.220644 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:51.220856 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 20:15:51.221149 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":363,"threads_started":1}
I20260923 20:15:51.224138 10881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd: No bootstrap required, opened a new log
I20260923 20:15:51.226018 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:15:51.231803 10881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb32afe32e64f1789c980b0c38967cd" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40415 } }
I20260923 20:15:51.232141 10881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:51.232254 10881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb32afe32e64f1789c980b0c38967cd, State: Initialized, Role: FOLLOWER
I20260923 20:15:51.232932 10881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb32afe32e64f1789c980b0c38967cd" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40415 } }
I20260923 20:15:51.233173 10881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:51.233322 10881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:51.233489 10881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:51.235700 10881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb32afe32e64f1789c980b0c38967cd" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40415 } }
I20260923 20:15:51.236171 10881 leader_election.cc:302] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eb32afe32e64f1789c980b0c38967cd; no voters:
I20260923 20:15:51.236790 10881 leader_election.cc:288] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:51.236946 10886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:51.237619 10886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [term 1 LEADER]: Becoming Leader. State: Replica: 5eb32afe32e64f1789c980b0c38967cd, State: Running, Role: LEADER
I20260923 20:15:51.238184 10886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb32afe32e64f1789c980b0c38967cd" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40415 } }
I20260923 20:15:51.239003 10881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:51.241087 10888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eb32afe32e64f1789c980b0c38967cd. Latest consensus state: current_term: 1 leader_uuid: "5eb32afe32e64f1789c980b0c38967cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb32afe32e64f1789c980b0c38967cd" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40415 } } }
I20260923 20:15:51.241521 10888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:51.242303 10887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eb32afe32e64f1789c980b0c38967cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb32afe32e64f1789c980b0c38967cd" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40415 } } }
I20260923 20:15:51.242702 10887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:51.246232 10895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:51.250211 10895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:51.258201 10895 catalog_manager.cc:1347] Generated new cluster ID: de7bb7c71cb046ecb0a822a8803b6bce
I20260923 20:15:51.258311 10895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:51.274700 10895 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:51.276145 10895 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:51.283133 10895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5eb32afe32e64f1789c980b0c38967cd: Generated new TSK 0
I20260923 20:15:51.283876 10895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:15:51.304248 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:51.295990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44508 (local address 127.5.83.126:40415)
0923 20:15:51.296256 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:51.296288 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:51.297020 (+ 732us) server_negotiation.cc:407] Connection header received
0923 20:15:51.297283 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:51.297287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:51.297352 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:51.297471 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:15:51.297979 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:51.298687 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:51.300691 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:51.301128 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:51.301223 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:51.301774 (+ 551us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:15:51.303569 (+ 1795us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 20:15:51.303898 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44508: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":118}
W20260923 20:15:51.304371 10884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44508: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 20:15:51.304420 10910 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:15:51.295846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:40415 (local address 127.0.0.1:44508)
0923 20:15:51.296902 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:51.296922 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:51.297073 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:51.297517 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:51.297522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:51.297535 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:15:51.297831 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:51.297844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:51.298826 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:51.298832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:51.300559 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:51.300576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:51.301318 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:51.301325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:51.301510 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:51.301555 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:15:51.303753 (+ 2198us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 20:15:51.303988 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.83.126:40415: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36575/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":892,"thread_start_us":668,"threads_started":1}
I20260923 20:15:51.306032 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 10815
I20260923 20:15:51.307039 10876 generic_service.cc:196] Checking for leaks (request via RPC)
E20260923 20:15:51.316413 10911 webserver.cc:533] Webserver: sslize failed:
E20260923 20:15:51.316514 10911 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 20:15:51.398417 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 10815
2026-09-23T20:15:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (700 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T20:15:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:51Z Disabled control of system clock
I20260923 20:15:51.483688 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:32969
--webserver_interface=127.5.83.126
--webserver_port=0
--builtin_ntp_servers=127.5.83.84:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:32969
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/testchainca.cert with env {}
W20260923 20:15:51.808164 10916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:51.808454 10916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:51.808609 10916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:51.815845 10916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:51.815959 10916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:51.815999 10916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:51.816051 10916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:51.816092 10916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 20:15:51.816155 10916 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 20:15:51.816193 10916 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 20:15:51.826340 10916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:32969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:51.827626 10916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:51.828924 10916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:51.840996 10924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:51.841084 10922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:51.841037 10921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:51.842341 10916 server_base.cc:1034] running on GCE node
I20260923 20:15:51.843178 10916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:51.844628 10916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:51.845845 10916 hybrid_clock.cc:648] HybridClock initialized: now 1790194551845809 us; error 29 us; skew 500 ppm
I20260923 20:15:51.846240 10916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:15:51.848452 10916 webserver.cc:460] Webserver started at http://127.5.83.126:35491/ using document root <none> and password file <none>
I20260923 20:15:51.849205 10916 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:51.849296 10916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:51.849649 10916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:51.852510 10916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "c20a06d586364375afe4daac6f179b62"
format_stamp: "Formatted at 2026-09-23 20:15:51 on dist-test-slave-vl24"
server_key: "edde5b059b8ab61cf28852e5dec0e20b"
server_key_iv: "6a51e1c697528f32c99f3143f8887ad4"
server_key_version: "encryptionkey@0"
I20260923 20:15:51.853369 10916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "c20a06d586364375afe4daac6f179b62"
format_stamp: "Formatted at 2026-09-23 20:15:51 on dist-test-slave-vl24"
server_key: "edde5b059b8ab61cf28852e5dec0e20b"
server_key_iv: "6a51e1c697528f32c99f3143f8887ad4"
server_key_version: "encryptionkey@0"
I20260923 20:15:51.858649 10916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:15:51.862246 10930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:51.863565 10916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:15:51.863746 10916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "c20a06d586364375afe4daac6f179b62"
format_stamp: "Formatted at 2026-09-23 20:15:51 on dist-test-slave-vl24"
server_key: "edde5b059b8ab61cf28852e5dec0e20b"
server_key_iv: "6a51e1c697528f32c99f3143f8887ad4"
server_key_version: "encryptionkey@0"
I20260923 20:15:51.863961 10916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:51.878330 10916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:51.879122 10916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:51.879392 10916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:51.898375 10916 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:32969
I20260923 20:15:51.898387 10981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:32969 every 8 connection(s)
I20260923 20:15:51.899773 10916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:51.903313 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 10916
I20260923 20:15:51.903546 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:51.903935 10982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:51.904417 5453 external_mini_cluster.cc:1428] Setting key c7f4712fb1a09c36d8a278cff4eac821
I20260923 20:15:51.916821 10982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62: Bootstrap starting.
I20260923 20:15:51.917826 10985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:51.906342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.5.83.126:32969)
0923 20:15:51.907339 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:51.907349 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:51.907390 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:15:51.909096 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:51.909120 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:51.909618 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:51.910034 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:51.910635 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:51.911922 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:51.913318 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:51.914019 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:51.914254 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:51.915474 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:51.915481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:51.915494 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:15:51.916027 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:51.916109 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:15:51.916130 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:51.916257 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:51.916493 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 20:15:51.916837 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":339,"threads_started":1}
I20260923 20:15:51.920910 10982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:51.922431 10982 log.cc:824] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:51.925410 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:15:51.926156 10982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62: No bootstrap required, opened a new log
I20260923 20:15:51.934162 10982 raft_consensus.cc:348] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20a06d586364375afe4daac6f179b62" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 32969 } }
I20260923 20:15:51.934499 10982 raft_consensus.cc:374] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:51.934573 10982 raft_consensus.cc:729] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20a06d586364375afe4daac6f179b62, State: Initialized, Role: FOLLOWER
I20260923 20:15:51.935133 10982 consensus_queue.cc:260] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20a06d586364375afe4daac6f179b62" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 32969 } }
I20260923 20:15:51.935385 10982 raft_consensus.cc:388] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:51.935529 10982 raft_consensus.cc:482] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:51.935725 10982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:51.938020 10982 raft_consensus.cc:504] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20a06d586364375afe4daac6f179b62" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 32969 } }
I20260923 20:15:51.938529 10982 leader_election.cc:302] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c20a06d586364375afe4daac6f179b62; no voters:
I20260923 20:15:51.939148 10982 leader_election.cc:288] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:51.939405 10987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:51.940157 10987 raft_consensus.cc:686] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [term 1 LEADER]: Becoming Leader. State: Replica: c20a06d586364375afe4daac6f179b62, State: Running, Role: LEADER
I20260923 20:15:51.940709 10987 consensus_queue.cc:237] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20a06d586364375afe4daac6f179b62" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 32969 } }
I20260923 20:15:51.941336 10982 sys_catalog.cc:564] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:51.943686 10989 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c20a06d586364375afe4daac6f179b62. Latest consensus state: current_term: 1 leader_uuid: "c20a06d586364375afe4daac6f179b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20a06d586364375afe4daac6f179b62" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 32969 } } }
I20260923 20:15:51.944015 10989 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:51.944392 10988 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c20a06d586364375afe4daac6f179b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20a06d586364375afe4daac6f179b62" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 32969 } } }
I20260923 20:15:51.944845 10988 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:51.946204 10992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:51.951418 10992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:51.960839 10992 catalog_manager.cc:1347] Generated new cluster ID: 9a20230903d143c2a763e6e992f58e46
I20260923 20:15:51.961032 10992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:51.973397 10992 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:51.974926 10992 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:51.989778 10992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c20a06d586364375afe4daac6f179b62: Generated new TSK 0
I20260923 20:15:51.990664 10992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:15:52.030550 10985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:52.004755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.5.83.126:32969)
0923 20:15:52.004955 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:52.004964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:52.005331 (+ 367us) server_negotiation.cc:407] Connection header received
0923 20:15:52.005608 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:52.005612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:52.005684 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:52.005858 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0923 20:15:52.006369 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:52.007125 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:52.008485 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:52.008980 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:52.009115 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:52.009631 (+ 516us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 20:15:52.029809 (+ 20178us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 20:15:52.030230 (+ 421us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36862: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 20:15:52.030717 10985 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36862: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 20:15:52.030823 11011 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:15:52.004634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:32969 (local address 127.0.0.1:36862)
0923 20:15:52.005224 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:52.005239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:52.005407 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:52.005873 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:52.005878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:52.005890 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0923 20:15:52.006214 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:52.006223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:52.007279 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:52.007285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:52.008342 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:52.008354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:52.009159 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:52.009167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:52.009323 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:52.009356 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 20:15:52.030110 (+ 20754us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 20:15:52.030390 (+ 280us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.83.126:32969: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":325,"threads_started":1}
I20260923 20:15:52.031874 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 10916
I20260923 20:15:52.033104 10977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:52.127292 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 10916
2026-09-23T20:15:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (708 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (412 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (410 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:15:53 dist-test-slave-vl24 krb5kdc[11026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:15:53 dist-test-slave-vl24 krb5kdc[11026](info): set up 2 sockets
Sep 23 20:15:53 dist-test-slave-vl24 krb5kdc[11026](info): commencing operation
krb5kdc: starting...
W20260923 20:15:54.070628 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:15:54 dist-test-slave-vl24 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:15:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:15:54.285501 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:42871
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:42871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:54.611840 11039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:54.612128 11039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:54.612217 11039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:54.619678 11039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:54.619803 11039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:54.619850 11039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:54.619900 11039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:54.619946 11039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:15:54.630344 11039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:42871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:42871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:54.631709 11039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:54.633031 11039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:54.645450 11044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:54.645500 11045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:54.645649 11047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:54.646358 11039 server_base.cc:1034] running on GCE node
I20260923 20:15:54.647148 11039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:54.648736 11039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:54.649974 11039 hybrid_clock.cc:648] HybridClock initialized: now 1790194554649936 us; error 54 us; skew 500 ppm
I20260923 20:15:54.650331 11039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:54 dist-test-slave-vl24 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194554, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:54.653563 11039 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:15:54.655197 11039 webserver.cc:460] Webserver started at http://127.5.83.126:43687/ using document root <none> and password file <none>
I20260923 20:15:54.655921 11039 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:54.656073 11039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:54.656407 11039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:54.659474 11039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "d957363eb8a8428b95e8eee9e6ec3634"
format_stamp: "Formatted at 2026-09-23 20:15:54 on dist-test-slave-vl24"
server_key: "2aedb4ddda05c318efe45bc015eb5077"
server_key_iv: "48cbf600f02176d8ed6a17e0d3b6ecca"
server_key_version: "encryptionkey@0"
I20260923 20:15:54.660281 11039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "d957363eb8a8428b95e8eee9e6ec3634"
format_stamp: "Formatted at 2026-09-23 20:15:54 on dist-test-slave-vl24"
server_key: "2aedb4ddda05c318efe45bc015eb5077"
server_key_iv: "48cbf600f02176d8ed6a17e0d3b6ecca"
server_key_version: "encryptionkey@0"
I20260923 20:15:54.665508 11039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:15:54.668942 11054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:54.670250 11039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:15:54.670442 11039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "d957363eb8a8428b95e8eee9e6ec3634"
format_stamp: "Formatted at 2026-09-23 20:15:54 on dist-test-slave-vl24"
server_key: "2aedb4ddda05c318efe45bc015eb5077"
server_key_iv: "48cbf600f02176d8ed6a17e0d3b6ecca"
server_key_version: "encryptionkey@0"
I20260923 20:15:54.670670 11039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:54.702652 11039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:54.707072 11039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:54.707384 11039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:54.726164 11039 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:42871
I20260923 20:15:54.726199 11105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:42871 every 8 connection(s)
I20260923 20:15:54.727566 11039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:54.731431 11106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:54.737349 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 11039
I20260923 20:15:54.737591 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:54.738386 5453 external_mini_cluster.cc:1428] Setting key 00c79ef7f02fe932c5ce71ea3fc17a5d
I20260923 20:15:54.742861 11106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634: Bootstrap starting.
I20260923 20:15:54.746721 11106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:54.747925 11106 log.cc:824] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:54.751571 11106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634: No bootstrap required, opened a new log
Sep 23 20:15:54 dist-test-slave-vl24 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:54.759850 11106 raft_consensus.cc:348] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d957363eb8a8428b95e8eee9e6ec3634" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42871 } }
I20260923 20:15:54.760267 11106 raft_consensus.cc:374] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:54.760388 11106 raft_consensus.cc:729] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d957363eb8a8428b95e8eee9e6ec3634, State: Initialized, Role: FOLLOWER
I20260923 20:15:54.761158 11106 consensus_queue.cc:260] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d957363eb8a8428b95e8eee9e6ec3634" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42871 } }
I20260923 20:15:54.761459 11106 raft_consensus.cc:388] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:54.761616 11106 raft_consensus.cc:482] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:54.761825 11106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:54.762456 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:54.740565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.5.83.126:42871)
0923 20:15:54.741383 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:54.741396 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:54.741441 (+ 45us) server_negotiation.cc:407] Connection header received
0923 20:15:54.743221 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:54.743250 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:54.743702 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:54.744016 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:54.745893 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:54.747316 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:54.749111 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:54.749911 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:54.750055 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:54.754835 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:54.754842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:54.754859 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:54.754907 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:54.758155 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:54.758885 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:54.758892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:54.758899 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:54.759073 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:54.759464 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:54.759469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:54.759471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:54.760024 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:54.760338 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:54.760796 (+ 458us) server_negotiation.cc:299] Negotiation successful
0923 20:15:54.761225 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":395,"threads_started":1}
I20260923 20:15:54.764259 11106 raft_consensus.cc:504] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d957363eb8a8428b95e8eee9e6ec3634" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42871 } }
I20260923 20:15:54.764846 11106 leader_election.cc:302] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d957363eb8a8428b95e8eee9e6ec3634; no voters:
I20260923 20:15:54.765528 11106 leader_election.cc:288] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:54.765632 11111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:54.766376 11111 raft_consensus.cc:686] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [term 1 LEADER]: Becoming Leader. State: Replica: d957363eb8a8428b95e8eee9e6ec3634, State: Running, Role: LEADER
I20260923 20:15:54.766916 11111 consensus_queue.cc:237] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d957363eb8a8428b95e8eee9e6ec3634" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42871 } }
I20260923 20:15:54.767709 11106 sys_catalog.cc:564] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:54.769837 11112 sys_catalog.cc:455] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d957363eb8a8428b95e8eee9e6ec3634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d957363eb8a8428b95e8eee9e6ec3634" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42871 } } }
I20260923 20:15:54.770296 11112 sys_catalog.cc:458] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:54.771173 11113 sys_catalog.cc:455] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d957363eb8a8428b95e8eee9e6ec3634. Latest consensus state: current_term: 1 leader_uuid: "d957363eb8a8428b95e8eee9e6ec3634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d957363eb8a8428b95e8eee9e6ec3634" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42871 } } }
I20260923 20:15:54.771519 11113 sys_catalog.cc:458] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:54.772473 11118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:54.777907 11118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:54.786918 11118 catalog_manager.cc:1347] Generated new cluster ID: 4e49c96fcd8a4566bcd805ab95aa5fef
I20260923 20:15:54.787058 11118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:54.824365 11118 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:54.825731 11118 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:54.831771 11118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d957363eb8a8428b95e8eee9e6ec3634: Generated new TSK 0
I20260923 20:15:54.832620 11118 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:15:54.896517 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42871
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:55.252947 11132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:55.253226 11132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:55.253328 11132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:55.260754 11132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:55.260867 11132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:55.261039 11132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:15:55.272150 11132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:55.273559 11132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:55.275111 11132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:55.288339 11138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:55.288590 11137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:55.288352 11140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:55.289336 11132 server_base.cc:1034] running on GCE node
I20260923 20:15:55.289952 11132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:55.290898 11132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:55.292150 11132 hybrid_clock.cc:648] HybridClock initialized: now 1790194555292114 us; error 44 us; skew 500 ppm
I20260923 20:15:55.292495 11132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:55 dist-test-slave-vl24 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194555, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:55.295933 11132 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:15:55.297444 11132 webserver.cc:460] Webserver started at http://127.5.83.65:46053/ using document root <none> and password file <none>
I20260923 20:15:55.298130 11132 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:55.298273 11132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:55.298573 11132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:55.301964 11132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "fbabcb861c3a4cf0b8c508e7d022a60d"
format_stamp: "Formatted at 2026-09-23 20:15:55 on dist-test-slave-vl24"
server_key: "6d785086795171bd617fa99d0b567d56"
server_key_iv: "1a9bf25a380e3812834b4fca64c69b34"
server_key_version: "encryptionkey@0"
I20260923 20:15:55.302757 11132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "fbabcb861c3a4cf0b8c508e7d022a60d"
format_stamp: "Formatted at 2026-09-23 20:15:55 on dist-test-slave-vl24"
server_key: "6d785086795171bd617fa99d0b567d56"
server_key_iv: "1a9bf25a380e3812834b4fca64c69b34"
server_key_version: "encryptionkey@0"
I20260923 20:15:55.307898 11132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:15:55.311753 11147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:55.313196 11132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:15:55.313380 11132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "fbabcb861c3a4cf0b8c508e7d022a60d"
format_stamp: "Formatted at 2026-09-23 20:15:55 on dist-test-slave-vl24"
server_key: "6d785086795171bd617fa99d0b567d56"
server_key_iv: "1a9bf25a380e3812834b4fca64c69b34"
server_key_version: "encryptionkey@0"
I20260923 20:15:55.313619 11132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:55.328960 11132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:55.334077 11132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:55.334390 11132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:55.335397 11132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:55.336875 11132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:55.336978 11132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:55.337133 11132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:55.337210 11132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:55.370639 11132 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:37205
I20260923 20:15:55.370697 11259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:37205 every 8 connection(s)
I20260923 20:15:55.371935 11132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:15:55.381798 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 11132
I20260923 20:15:55.382035 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:15:55.382638 5453 external_mini_cluster.cc:1428] Setting key 47527aac537b5b974b5583b7217c577c
Sep 23 20:15:55 dist-test-slave-vl24 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194555, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:55.394883 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:55.374800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:48751 (local address 127.5.83.126:42871)
0923 20:15:55.375287 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:55.375294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:55.376214 (+ 920us) server_negotiation.cc:407] Connection header received
0923 20:15:55.378165 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:55.378169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:55.378237 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:55.378334 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:55.379870 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:55.380654 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:55.382396 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:55.382866 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:55.383020 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:55.387342 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:55.387351 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:55.387359 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:55.387406 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:55.391206 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:55.392258 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:55.392263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:55.392265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:55.392401 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:55.392956 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:55.392960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:55.392962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:55.393251 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:55.393424 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:55.394212 (+ 788us) server_negotiation.cc:299] Negotiation successful
0923 20:15:55.394453 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":243,"threads_started":1}
I20260923 20:15:55.395907 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:55.375256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42871 (local address 127.5.83.65:48751)
0923 20:15:55.376056 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:55.376096 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:55.377888 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:55.378580 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:55.378588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:55.379125 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:55.379712 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:55.379730 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:55.380853 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:55.380857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:55.382247 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:55.382259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:55.383114 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:55.383123 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:55.383420 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:55.384200 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:55.384223 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:55.387059 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:55.391379 (+ 4320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:55.391393 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:55.391423 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:55.392092 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:55.392551 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:55.392560 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:55.392565 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:55.392796 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:55.393393 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:55.393403 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:55.393780 (+ 377us) client_negotiation.cc:769] Sending connection context
0923 20:15:55.394095 (+ 315us) client_negotiation.cc:240] Negotiation successful
0923 20:15:55.394674 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":317,"threads_started":1}
I20260923 20:15:55.397643 11260 heartbeater.cc:347] Connected to a master server at 127.5.83.126:42871
I20260923 20:15:55.397986 11260 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:55.398870 11260 heartbeater.cc:511] Master 127.5.83.126:42871 requested a full tablet report, sending...
I20260923 20:15:55.401490 11071 ts_manager.cc:194] Registered new tserver with Master: fbabcb861c3a4cf0b8c508e7d022a60d (127.5.83.65:37205)
I20260923 20:15:55.403849 11071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:48751
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:15:55.435535 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42871
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:55.768343 11266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:55.768762 11266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:55.768932 11266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:55.777382 11266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:55.777515 11266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:55.777707 11266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:15:55.789045 11266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:55.790512 11266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:55.791927 11266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:55.807116 11272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:55.807163 11271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:55.807144 11274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:55.813489 11266 server_base.cc:1034] running on GCE node
I20260923 20:15:55.814262 11266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:55.815462 11266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:55.816712 11266 hybrid_clock.cc:648] HybridClock initialized: now 1790194555816667 us; error 67 us; skew 500 ppm
I20260923 20:15:55.817150 11266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:55 dist-test-slave-vl24 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194555, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:55.821022 11266 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:15:55.822820 11266 webserver.cc:460] Webserver started at http://127.5.83.66:34913/ using document root <none> and password file <none>
I20260923 20:15:55.823577 11266 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:55.823690 11266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:55.823966 11266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:55.827309 11266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "6ea8c154c374423cac99ec459f7406ca"
format_stamp: "Formatted at 2026-09-23 20:15:55 on dist-test-slave-vl24"
server_key: "2a971b7f7678228bfc4b218bcf1ddbcd"
server_key_iv: "a916f469722a739b0272fec8a3aa8ad6"
server_key_version: "encryptionkey@0"
I20260923 20:15:55.828194 11266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "6ea8c154c374423cac99ec459f7406ca"
format_stamp: "Formatted at 2026-09-23 20:15:55 on dist-test-slave-vl24"
server_key: "2a971b7f7678228bfc4b218bcf1ddbcd"
server_key_iv: "a916f469722a739b0272fec8a3aa8ad6"
server_key_version: "encryptionkey@0"
I20260923 20:15:55.834154 11266 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.001s
I20260923 20:15:55.838300 11281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:55.839983 11266 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:15:55.840195 11266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "6ea8c154c374423cac99ec459f7406ca"
format_stamp: "Formatted at 2026-09-23 20:15:55 on dist-test-slave-vl24"
server_key: "2a971b7f7678228bfc4b218bcf1ddbcd"
server_key_iv: "a916f469722a739b0272fec8a3aa8ad6"
server_key_version: "encryptionkey@0"
I20260923 20:15:55.840453 11266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:55.878115 11266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:55.882266 11266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:55.882565 11266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:55.883623 11266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:55.884985 11266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:55.885072 11266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:55.885267 11266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:55.885317 11266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:55.919535 11266 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:39853
I20260923 20:15:55.919657 11393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:39853 every 8 connection(s)
I20260923 20:15:55.920974 11266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:15:55.927645 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 11266
I20260923 20:15:55.927838 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:15:55.928391 5453 external_mini_cluster.cc:1428] Setting key 00bd31555c5208a1d6610ba1e537f1e7
Sep 23 20:15:55 dist-test-slave-vl24 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194555, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:55.943367 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:55.923523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:50631 (local address 127.5.83.126:42871)
0923 20:15:55.923991 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:55.923995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:55.924837 (+ 842us) server_negotiation.cc:407] Connection header received
0923 20:15:55.926671 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:55.926676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:55.926772 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:55.926893 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:55.928964 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:55.929832 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:55.932146 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:55.932606 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:55.932710 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:55.936912 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:55.936917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:55.936921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:55.936953 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:55.939880 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:55.940950 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:55.940954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:55.940956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:55.941114 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:55.941644 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:55.941647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:55.941648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:55.941913 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:55.942021 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:55.942927 (+ 906us) server_negotiation.cc:299] Negotiation successful
0923 20:15:55.943103 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":240,"threads_started":1}
I20260923 20:15:55.945125 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:55.923926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42871 (local address 127.5.83.66:50631)
0923 20:15:55.924686 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:55.924723 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:55.926388 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:55.927128 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:55.927136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:55.927922 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:55.928762 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:55.928790 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:55.929977 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:55.929981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:55.931996 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:55.932015 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:55.932826 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:55.932835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:55.933179 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:55.933923 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:55.933958 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:55.936678 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:55.940065 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:55.940078 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:55.940099 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:55.940816 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:55.941239 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:55.941247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:55.941253 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:55.941528 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:55.942043 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:55.942054 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:55.942534 (+ 480us) client_negotiation.cc:769] Sending connection context
0923 20:15:55.942843 (+ 309us) client_negotiation.cc:240] Negotiation successful
0923 20:15:55.943486 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":291,"threads_started":1}
I20260923 20:15:55.946775 11394 heartbeater.cc:347] Connected to a master server at 127.5.83.126:42871
I20260923 20:15:55.947149 11394 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:55.948164 11394 heartbeater.cc:511] Master 127.5.83.126:42871 requested a full tablet report, sending...
I20260923 20:15:55.949797 11071 ts_manager.cc:194] Registered new tserver with Master: 6ea8c154c374423cac99ec459f7406ca (127.5.83.66:39853)
I20260923 20:15:55.950618 11071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:50631
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:15:55.979422 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42871
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:56.309510 11400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:56.309779 11400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:56.309949 11400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:56.317368 11400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:56.317479 11400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:56.317667 11400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:15:56.327924 11400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:56.329412 11400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:56.330755 11400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:56.343910 11405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:56.344045 11406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:56.343894 11408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:56.344971 11400 server_base.cc:1034] running on GCE node
I20260923 20:15:56.345726 11400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:56.346805 11400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:56.348057 11400 hybrid_clock.cc:648] HybridClock initialized: now 1790194556347987 us; error 72 us; skew 500 ppm
I20260923 20:15:56.348379 11400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:56.351575 11400 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:15:56.353158 11400 webserver.cc:460] Webserver started at http://127.5.83.67:33055/ using document root <none> and password file <none>
I20260923 20:15:56.353837 11400 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:56.353979 11400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:56.354305 11400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:56.357229 11400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "d481eb4a07d048b0bab6749ffcabe20e"
format_stamp: "Formatted at 2026-09-23 20:15:56 on dist-test-slave-vl24"
server_key: "c16c9829297f69ed8bf6328b911519a2"
server_key_iv: "830b4673b240e4b6836c9646ce0bdbca"
server_key_version: "encryptionkey@0"
I20260923 20:15:56.358019 11400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "d481eb4a07d048b0bab6749ffcabe20e"
format_stamp: "Formatted at 2026-09-23 20:15:56 on dist-test-slave-vl24"
server_key: "c16c9829297f69ed8bf6328b911519a2"
server_key_iv: "830b4673b240e4b6836c9646ce0bdbca"
server_key_version: "encryptionkey@0"
I20260923 20:15:56.363173 11400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:15:56.366500 11415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:56.367771 11400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:15:56.367935 11400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "d481eb4a07d048b0bab6749ffcabe20e"
format_stamp: "Formatted at 2026-09-23 20:15:56 on dist-test-slave-vl24"
server_key: "c16c9829297f69ed8bf6328b911519a2"
server_key_iv: "830b4673b240e4b6836c9646ce0bdbca"
server_key_version: "encryptionkey@0"
I20260923 20:15:56.368222 11400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:56.394555 11400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:56.398940 11400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:56.399250 11400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:56.400287 11400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:56.401721 11400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:56.401829 11400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:56.401975 11400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:56.402048 11400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:56.407632 11260 heartbeater.cc:503] Master 127.5.83.126:42871 was elected leader, sending a full tablet report...
I20260923 20:15:56.435345 11400 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:40135
I20260923 20:15:56.435391 11527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:40135 every 8 connection(s)
I20260923 20:15:56.436584 11400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:15:56.440896 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 11400
I20260923 20:15:56.441098 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:15:56.441498 5453 external_mini_cluster.cc:1428] Setting key eb46b203035543c7a1dc18a1bb3f3388
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:15:56.457464 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:56.439113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:47733 (local address 127.5.83.126:42871)
0923 20:15:56.439301 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:56.439305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:56.440364 (+ 1059us) server_negotiation.cc:407] Connection header received
0923 20:15:56.442340 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:56.442346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:56.442480 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:56.442591 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:56.444149 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.445014 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.446733 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.447193 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.447329 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:56.450624 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:56.450629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:56.450633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:56.450662 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:56.454254 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.455052 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.455056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.455058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.455179 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.455636 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.455641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.455645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.455951 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:56.456118 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:56.456909 (+ 791us) server_negotiation.cc:299] Negotiation successful
0923 20:15:56.457152 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:15:56.458396 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:56.439502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42871 (local address 127.5.83.67:47733)
0923 20:15:56.440196 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:56.440231 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:56.442069 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:56.442808 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:56.442816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:56.443377 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:56.443978 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:56.443994 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.445150 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.445160 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:56.446581 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:56.446592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.447423 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.447429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:56.447633 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:56.448279 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:56.448298 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:56.450432 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:56.454469 (+ 4037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:56.454478 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:56.454495 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:56.454920 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:56.455291 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:56.455295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:56.455299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:56.455516 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:56.456091 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:56.456101 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:56.456425 (+ 324us) client_negotiation.cc:769] Sending connection context
0923 20:15:56.456772 (+ 347us) client_negotiation.cc:240] Negotiation successful
0923 20:15:56.457301 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":269,"threads_started":1}
I20260923 20:15:56.459733 11528 heartbeater.cc:347] Connected to a master server at 127.5.83.126:42871
I20260923 20:15:56.460039 11528 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:56.460794 11528 heartbeater.cc:511] Master 127.5.83.126:42871 requested a full tablet report, sending...
I20260923 20:15:56.462181 11071 ts_manager.cc:194] Registered new tserver with Master: d481eb4a07d048b0bab6749ffcabe20e (127.5.83.67:40135)
I20260923 20:15:56.462985 11071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:47733
I20260923 20:15:56.469260 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/test-admin.keytab.
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:56.497591 5453 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 20:15:56.502768 11535 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:15:56.500144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.5.83.65:37205)
0923 20:15:56.500874 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:56.500881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:56.500906 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:15:56.500972 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:56.500979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:56.501182 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:56.501350 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:56.502330 (+ 980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34494: BlockingRecv error: recv got EOF from 127.0.0.1:34494 (error 108)
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":386,"threads_started":1}
W20260923 20:15:56.502907 5453 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/krb5cc' of type FILE
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:56.504522 5453 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:56.506067 5453 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:56.521900 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:56.507702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.5.83.65:37205)
0923 20:15:56.507894 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:56.507900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:56.508106 (+ 206us) server_negotiation.cc:407] Connection header received
0923 20:15:56.508308 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:56.508312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:56.508368 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:56.508468 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:56.509441 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.510396 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.511645 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.512195 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.512333 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:56.515866 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:56.515877 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:56.515891 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:56.515924 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:56.519267 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.519895 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.519904 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.519906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.520041 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.520354 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.520357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.520359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.520773 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:56.521037 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:56.521359 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 20:15:56.521620 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 20:15:56.527563 11535 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:15:56.525857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34526 (local address 127.5.83.65:37205)
0923 20:15:56.526092 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:56.526099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:56.526326 (+ 227us) server_negotiation.cc:407] Connection header received
0923 20:15:56.526522 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:56.526526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:56.526591 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:56.526691 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:56.527350 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34526: BlockingRecv error: recv got EOF from 127.0.0.1:34526 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260923 20:15:56.528308 5453 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/krb5cc)
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:56.529983 5453 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:56.531384 5453 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:56.544968 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:56.532715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34532 (local address 127.5.83.65:37205)
0923 20:15:56.532911 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:56.532918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:56.533225 (+ 307us) server_negotiation.cc:407] Connection header received
0923 20:15:56.533341 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:56.533344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:56.533399 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:56.533474 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:56.534459 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.535134 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.536264 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.536743 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.536849 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:56.540192 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:56.540197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:56.540200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:56.540241 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:56.542601 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.543140 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.543143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.543146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.543251 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.543610 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.543612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.543614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.543916 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:56.544124 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:56.544382 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 20:15:56.544601 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260923 20:15:56.549420 11535 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:15:56.547775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.5.83.65:37205)
0923 20:15:56.547949 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:56.547961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:56.548119 (+ 158us) server_negotiation.cc:407] Connection header received
0923 20:15:56.548233 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:56.548237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:56.548296 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:56.548368 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:56.549179 (+ 811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34544: BlockingRecv error: recv got EOF from 127.0.0.1:34544 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 20:15:56.549588 5453 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790194506409421-5453-0/krb5kdc/krb5cc' of type FILE
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:56.551151 5453 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:56.552456 5453 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:15:56.565872 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:56.553810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.5.83.65:37205)
0923 20:15:56.554010 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:56.554015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:56.554169 (+ 154us) server_negotiation.cc:407] Connection header received
0923 20:15:56.554284 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:56.554288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:56.554342 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:56.554419 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:56.555326 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.556034 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.557255 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.557660 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.557751 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:56.561066 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:56.561071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:56.561074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:56.561104 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:56.563525 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.564151 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.564156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.564158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.564315 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.564720 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.564724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.564726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.565032 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:56.565263 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:56.565441 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 20:15:56.565618 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:15:56.567519 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 11132
I20260923 20:15:56.579703 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:56.568236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.5.83.65:37205)
0923 20:15:56.568401 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:56.568407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:56.568805 (+ 398us) server_negotiation.cc:407] Connection header received
0923 20:15:56.568995 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:56.569000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:56.569066 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:56.569164 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:56.570072 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.570739 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.571938 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.572355 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.572497 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:56.574689 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:56.574695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:56.574699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:56.574736 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:56.577207 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.577838 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.577845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.577848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.577984 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.578339 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.578345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.578355 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.578618 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:56.578884 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:56.579180 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 20:15:56.579388 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 20:15:56.580277 11255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:56.663609 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 11132
I20260923 20:15:56.680374 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 11266
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:15:56.695459 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:56.681302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.5.83.66:39853)
0923 20:15:56.681807 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:56.681815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:56.681839 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:15:56.681897 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:56.681904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:56.682163 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:56.682381 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:56.683404 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.684392 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.685692 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.686200 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.686333 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:56.690059 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:56.690069 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:56.690082 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:56.690119 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:56.692778 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.693437 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.693445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.693448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.693572 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.693922 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.693929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.693933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.694256 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:56.694563 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:56.694878 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 20:15:56.695140 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
I20260923 20:15:56.696765 11389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:56.780336 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 11266
I20260923 20:15:56.796849 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 11400
Sep 23 20:15:56 dist-test-slave-vl24 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:15:56.812376 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:56.797976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60354 (local address 127.5.83.67:40135)
0923 20:15:56.798602 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:56.798609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:56.798638 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:15:56.798696 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:56.798702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:56.798918 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:56.799039 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:56.800126 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.801228 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.802475 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:56.803033 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:56.803136 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:56.807107 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:56.807118 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:56.807132 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:56.807165 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:56.809988 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.810598 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.810605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.810607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.810720 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:56.811042 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:56.811045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:56.811047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:56.811361 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:56.811604 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:56.811809 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 20:15:56.812069 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":322,"threads_started":1}
I20260923 20:15:56.813653 11523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:56.897544 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 11400
I20260923 20:15:56.913643 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 11039
I20260923 20:15:56.914959 11101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:15:57.022102 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 11039
2026-09-23T20:15:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4077 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:15:57 dist-test-slave-vl24 krb5kdc[11555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:15:57 dist-test-slave-vl24 krb5kdc[11555](info): set up 2 sockets
Sep 23 20:15:57 dist-test-slave-vl24 krb5kdc[11555](info): commencing operation
krb5kdc: starting...
W20260923 20:15:58.115118 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:15:58 dist-test-slave-vl24 krb5kdc[11555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:15:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:15:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.126.keytab.
Entry for principal oryx/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.126.keytab.
I20260923 20:15:58.332173 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:42939
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.126.keytab
--principal=oryx/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:42939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:58.657783 11568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:58.658093 11568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:58.658262 11568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:58.665799 11568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:15:58.665911 11568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:58.666000 11568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:58.666111 11568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:15:58.666174 11568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:15:58.676028 11568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:42939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.126.keytab
--principal=oryx/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:42939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:58.677415 11568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:58.678735 11568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:58.690542 11573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:58.690513 11574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:58.690474 11576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:58.691939 11568 server_base.cc:1034] running on GCE node
I20260923 20:15:58.692736 11568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:58.694074 11568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:58.695274 11568 hybrid_clock.cc:648] HybridClock initialized: now 1790194558695221 us; error 52 us; skew 500 ppm
I20260923 20:15:58.695670 11568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:58 dist-test-slave-vl24 krb5kdc[11555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194558, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:58.702169 11568 init.cc:377] Logged in from keytab as oryx/127.5.83.126@KRBTEST.COM (short username oryx)
I20260923 20:15:58.703644 11568 webserver.cc:460] Webserver started at http://127.5.83.126:38759/ using document root <none> and password file <none>
I20260923 20:15:58.704309 11568 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:58.704437 11568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:58.704797 11568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:58.707722 11568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "bbcb22e21bc74c539336c817d2b64b05"
format_stamp: "Formatted at 2026-09-23 20:15:58 on dist-test-slave-vl24"
server_key: "ac357be535c655eee4a4e79faf37bdf2"
server_key_iv: "6d569269209056840bbbec0d8249a914"
server_key_version: "encryptionkey@0"
I20260923 20:15:58.708469 11568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "bbcb22e21bc74c539336c817d2b64b05"
format_stamp: "Formatted at 2026-09-23 20:15:58 on dist-test-slave-vl24"
server_key: "ac357be535c655eee4a4e79faf37bdf2"
server_key_iv: "6d569269209056840bbbec0d8249a914"
server_key_version: "encryptionkey@0"
I20260923 20:15:58.713551 11568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:15:58.716929 11583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:58.718201 11568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:15:58.718392 11568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "bbcb22e21bc74c539336c817d2b64b05"
format_stamp: "Formatted at 2026-09-23 20:15:58 on dist-test-slave-vl24"
server_key: "ac357be535c655eee4a4e79faf37bdf2"
server_key_iv: "6d569269209056840bbbec0d8249a914"
server_key_version: "encryptionkey@0"
I20260923 20:15:58.718595 11568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:58.740281 11568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:58.744366 11568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:58.744710 11568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:58.763058 11568 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:42939
I20260923 20:15:58.763108 11634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:42939 every 8 connection(s)
I20260923 20:15:58.764415 11568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:15:58.768173 11635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:15:58.772879 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 11568
I20260923 20:15:58.773125 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:15:58.773741 5453 external_mini_cluster.cc:1428] Setting key 861f51cf1fec7fc4ce8ecdb5851d97d8
I20260923 20:15:58.778949 11635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05: Bootstrap starting.
I20260923 20:15:58.782878 11635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05: Neither blocks nor log segments found. Creating new log.
I20260923 20:15:58.784185 11635 log.cc:824] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05: Log is configured to *not* fsync() on all Append() calls
I20260923 20:15:58.787878 11635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05: No bootstrap required, opened a new log
Sep 23 20:15:58 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:15:58.797634 11635 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } }
I20260923 20:15:58.798163 11635 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:15:58.798264 11635 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbcb22e21bc74c539336c817d2b64b05, State: Initialized, Role: FOLLOWER
I20260923 20:15:58.798686 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:58.775804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45834 (local address 127.5.83.126:42939)
0923 20:15:58.776749 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:58.776764 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:58.776805 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:15:58.778454 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:58.778483 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:58.778867 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:58.779255 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:58.781143 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:58.782458 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:58.784317 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:58.785441 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:58.785655 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:58.790257 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:58.790265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:58.790279 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:58.790312 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:58.794206 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:58.794893 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:58.794904 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:58.794917 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:58.795122 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:58.795458 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:58.795464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:58.795467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:58.796187 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:58.796465 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:58.796825 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 20:15:58.797235 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":416,"threads_started":1}
I20260923 20:15:58.798907 11635 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } }
I20260923 20:15:58.799197 11635 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:15:58.799357 11635 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:15:58.799544 11635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:15:58.802131 11635 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } }
I20260923 20:15:58.802640 11635 leader_election.cc:302] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbcb22e21bc74c539336c817d2b64b05; no voters:
I20260923 20:15:58.803229 11635 leader_election.cc:288] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:15:58.803354 11640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:15:58.804035 11640 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 1 LEADER]: Becoming Leader. State: Replica: bbcb22e21bc74c539336c817d2b64b05, State: Running, Role: LEADER
I20260923 20:15:58.804627 11640 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } }
I20260923 20:15:58.805430 11635 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:15:58.807583 11642 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbcb22e21bc74c539336c817d2b64b05. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } }
I20260923 20:15:58.807571 11641 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } }
I20260923 20:15:58.808023 11642 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:58.808023 11641 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: This master's current role is: LEADER
I20260923 20:15:58.809971 11647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:15:58.814332 11647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:15:58.823283 11647 catalog_manager.cc:1347] Generated new cluster ID: 8028ca1dcc2a4dd2a2f380dd6787ff86
I20260923 20:15:58.823385 11647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:15:58.849153 11647 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:15:58.851075 11647 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:15:58.857719 11647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05: Generated new TSK 0
I20260923 20:15:58.858734 11647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab.
Entry for principal oryx/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab.
I20260923 20:15:58.912832 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42939
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab
--principal=oryx/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:59.244653 11661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:59.244933 11661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:59.245134 11661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:59.252300 11661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:59.252413 11661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:59.252856 11661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:15:59.262894 11661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab
--principal=oryx/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:59.264304 11661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:59.265743 11661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:59.278842 11666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:59.278842 11669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:59.278878 11667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:59.279595 11661 server_base.cc:1034] running on GCE node
I20260923 20:15:59.280210 11661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:59.281193 11661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:59.282411 11661 hybrid_clock.cc:648] HybridClock initialized: now 1790194559282360 us; error 46 us; skew 500 ppm
I20260923 20:15:59.282716 11661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:59 dist-test-slave-vl24 krb5kdc[11555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194559, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:59.285800 11661 init.cc:377] Logged in from keytab as oryx/127.5.83.65@KRBTEST.COM (short username oryx)
I20260923 20:15:59.287305 11661 webserver.cc:460] Webserver started at http://127.5.83.65:39417/ using document root <none> and password file <none>
I20260923 20:15:59.287993 11661 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:59.288134 11661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:59.288432 11661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:59.291345 11661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "8eb6ee177604473fa83b0371abc96e51"
format_stamp: "Formatted at 2026-09-23 20:15:59 on dist-test-slave-vl24"
server_key: "68177ef81ab7876d87c5416b5de73720"
server_key_iv: "aedd558d9b885edf3c0098de4f243e62"
server_key_version: "encryptionkey@0"
I20260923 20:15:59.292160 11661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "8eb6ee177604473fa83b0371abc96e51"
format_stamp: "Formatted at 2026-09-23 20:15:59 on dist-test-slave-vl24"
server_key: "68177ef81ab7876d87c5416b5de73720"
server_key_iv: "aedd558d9b885edf3c0098de4f243e62"
server_key_version: "encryptionkey@0"
I20260923 20:15:59.297627 11661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260923 20:15:59.301247 11676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:59.302692 11661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:15:59.302912 11661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "8eb6ee177604473fa83b0371abc96e51"
format_stamp: "Formatted at 2026-09-23 20:15:59 on dist-test-slave-vl24"
server_key: "68177ef81ab7876d87c5416b5de73720"
server_key_iv: "aedd558d9b885edf3c0098de4f243e62"
server_key_version: "encryptionkey@0"
I20260923 20:15:59.303148 11661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:59.324484 11661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:59.328696 11661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:59.329013 11661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:59.329988 11661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:59.331372 11661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:59.331482 11661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:59.331640 11661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:59.331712 11661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:59.365746 11661 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:40173
I20260923 20:15:59.365834 11788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:40173 every 8 connection(s)
I20260923 20:15:59.367079 11661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:15:59.375111 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 11661
I20260923 20:15:59.375357 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:15:59.376217 5453 external_mini_cluster.cc:1428] Setting key 423d54d2309dad47adef6b4177cd1d0a
Sep 23 20:15:59 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194559, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:15:59.390357 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:59.369844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:34065 (local address 127.5.83.126:42939)
0923 20:15:59.370320 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:59.370325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:59.371328 (+ 1003us) server_negotiation.cc:407] Connection header received
0923 20:15:59.373402 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:59.373410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:59.373500 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:59.373625 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:59.375473 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:59.376643 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:59.378523 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:59.379224 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:59.379412 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:59.383306 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:59.383317 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:59.383325 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:59.383375 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:59.386788 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:59.387796 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:59.387800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:59.387803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:59.387928 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:59.388487 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:59.388491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:59.388494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:59.388834 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:59.388953 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:59.389896 (+ 943us) server_negotiation.cc:299] Negotiation successful
0923 20:15:59.390097 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":230,"threads_started":1}
I20260923 20:15:59.391917 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:59.370431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.65:34065)
0923 20:15:59.371158 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:59.371200 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:59.373048 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:59.373955 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:59.373968 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:59.374629 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:59.375236 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:59.375290 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:59.376806 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:59.376810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:59.378369 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:59.378381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:59.379489 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:59.379495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:59.379717 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:59.380432 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:59.380460 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:59.383036 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:59.386956 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:59.386969 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:59.386989 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:59.387665 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:59.388079 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:59.388087 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:59.388094 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:59.388352 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:59.388960 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:59.388970 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:59.389473 (+ 503us) client_negotiation.cc:769] Sending connection context
0923 20:15:59.389818 (+ 345us) client_negotiation.cc:240] Negotiation successful
0923 20:15:59.390403 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":284,"threads_started":1}
I20260923 20:15:59.393589 11789 heartbeater.cc:347] Connected to a master server at 127.5.83.126:42939
I20260923 20:15:59.393896 11789 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:59.394670 11789 heartbeater.cc:511] Master 127.5.83.126:42939 requested a full tablet report, sending...
I20260923 20:15:59.396991 11600 ts_manager.cc:194] Registered new tserver with Master: 8eb6ee177604473fa83b0371abc96e51 (127.5.83.65:40173)
I20260923 20:15:59.398458 11600 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:34065
WARNING: no policy specified for oryx/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab.
Entry for principal oryx/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab.
I20260923 20:15:59.428305 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42939
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab
--principal=oryx/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:15:59.752758 11795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:15:59.753091 11795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:15:59.753258 11795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:15:59.760751 11795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:15:59.760897 11795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:15:59.761075 11795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:15:59.770793 11795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab
--principal=oryx/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:15:59.772066 11795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:15:59.773481 11795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:15:59.786604 11800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:59.786588 11801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:15:59.787002 11803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:15:59.787406 11795 server_base.cc:1034] running on GCE node
I20260923 20:15:59.787976 11795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:15:59.788968 11795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:15:59.790202 11795 hybrid_clock.cc:648] HybridClock initialized: now 1790194559790157 us; error 39 us; skew 500 ppm
I20260923 20:15:59.790511 11795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:15:59 dist-test-slave-vl24 krb5kdc[11555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194559, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:15:59.793612 11795 init.cc:377] Logged in from keytab as oryx/127.5.83.66@KRBTEST.COM (short username oryx)
I20260923 20:15:59.795157 11795 webserver.cc:460] Webserver started at http://127.5.83.66:44559/ using document root <none> and password file <none>
I20260923 20:15:59.795848 11795 fs_manager.cc:359] Metadata directory not provided
I20260923 20:15:59.795991 11795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:15:59.796304 11795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:15:59.799208 11795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "cff6b76a733b4b80ba43b8c45a19a60a"
format_stamp: "Formatted at 2026-09-23 20:15:59 on dist-test-slave-vl24"
server_key: "4fc4b03349410c5b210e09ded6634c31"
server_key_iv: "5c18621bef0fd071fdd3af2dff7ce66c"
server_key_version: "encryptionkey@0"
I20260923 20:15:59.799979 11795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "cff6b76a733b4b80ba43b8c45a19a60a"
format_stamp: "Formatted at 2026-09-23 20:15:59 on dist-test-slave-vl24"
server_key: "4fc4b03349410c5b210e09ded6634c31"
server_key_iv: "5c18621bef0fd071fdd3af2dff7ce66c"
server_key_version: "encryptionkey@0"
I20260923 20:15:59.805104 11795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:15:59.808861 11810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:59.810333 11795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:15:59.810534 11795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "cff6b76a733b4b80ba43b8c45a19a60a"
format_stamp: "Formatted at 2026-09-23 20:15:59 on dist-test-slave-vl24"
server_key: "4fc4b03349410c5b210e09ded6634c31"
server_key_iv: "5c18621bef0fd071fdd3af2dff7ce66c"
server_key_version: "encryptionkey@0"
I20260923 20:15:59.810740 11795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:15:59.840227 11795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:15:59.844184 11795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:15:59.844497 11795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:15:59.845485 11795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:15:59.846802 11795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:15:59.846899 11795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:59.847043 11795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:15:59.847146 11795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:15:59.880686 11795 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:45889
I20260923 20:15:59.880769 11922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:45889 every 8 connection(s)
I20260923 20:15:59.882005 11795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:15:59.890151 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 11795
I20260923 20:15:59.890368 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:15:59.890888 5453 external_mini_cluster.cc:1428] Setting key 65ee9a19636b26710b2423f4fc49661b
Sep 23 20:15:59 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194559, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:15:59.903867 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:59.884627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:35751 (local address 127.5.83.126:42939)
0923 20:15:59.884813 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:15:59.884818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:15:59.885940 (+ 1122us) server_negotiation.cc:407] Connection header received
0923 20:15:59.887903 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:15:59.887915 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:15:59.887988 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:15:59.888087 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:15:59.889719 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:59.890840 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:59.892654 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:15:59.893165 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:15:59.893288 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:59.897005 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:15:59.897015 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:15:59.897024 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:15:59.897070 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:15:59.900439 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:59.901492 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:59.901496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:59.901498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:59.901617 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:15:59.902114 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:15:59.902116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:15:59.902118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:15:59.902381 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:15:59.902482 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:15:59.903468 (+ 986us) server_negotiation.cc:299] Negotiation successful
0923 20:15:59.903638 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:15:59.905419 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:15:59.885098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.66:35751)
0923 20:15:59.885780 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0923 20:15:59.885820 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:15:59.887585 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:15:59.888340 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:15:59.888349 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:15:59.888980 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:15:59.889552 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:59.889570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:59.891074 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:59.891079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:59.892477 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:15:59.892489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:15:59.893389 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:15:59.893394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:15:59.893597 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:15:59.894295 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:15:59.894318 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:15:59.896774 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:15:59.900619 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:59.900630 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:59.900652 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:59.901361 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:59.901726 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:15:59.901734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:15:59.901741 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:15:59.901996 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:15:59.902496 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:15:59.902508 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:15:59.902953 (+ 445us) client_negotiation.cc:769] Sending connection context
0923 20:15:59.903282 (+ 329us) client_negotiation.cc:240] Negotiation successful
0923 20:15:59.903888 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":298,"threads_started":1}
I20260923 20:15:59.906960 11923 heartbeater.cc:347] Connected to a master server at 127.5.83.126:42939
I20260923 20:15:59.907305 11923 heartbeater.cc:464] Registering TS with master...
I20260923 20:15:59.908028 11923 heartbeater.cc:511] Master 127.5.83.126:42939 requested a full tablet report, sending...
I20260923 20:15:59.909583 11600 ts_manager.cc:194] Registered new tserver with Master: cff6b76a733b4b80ba43b8c45a19a60a (127.5.83.66:45889)
I20260923 20:15:59.910514 11600 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:35751
WARNING: no policy specified for oryx/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab.
Entry for principal oryx/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab.
I20260923 20:15:59.941442 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42939
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab
--principal=oryx/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:00.269734 11928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:00.270080 11928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:00.270263 11928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:00.277632 11928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:00.277786 11928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:00.277983 11928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:16:00.287834 11928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab
--principal=oryx/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:42939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:00.289268 11928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:00.290607 11928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:00.304404 11933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:00.304445 11936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:00.304502 11934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:00.305346 11928 server_base.cc:1034] running on GCE node
I20260923 20:16:00.305964 11928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:00.307001 11928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:00.308220 11928 hybrid_clock.cc:648] HybridClock initialized: now 1790194560308157 us; error 56 us; skew 500 ppm
I20260923 20:16:00.308679 11928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:00 dist-test-slave-vl24 krb5kdc[11555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194560, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:00.311934 11928 init.cc:377] Logged in from keytab as oryx/127.5.83.67@KRBTEST.COM (short username oryx)
I20260923 20:16:00.313433 11928 webserver.cc:460] Webserver started at http://127.5.83.67:42485/ using document root <none> and password file <none>
I20260923 20:16:00.314195 11928 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:00.314338 11928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:00.314654 11928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:00.317739 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "8ada6626f6f04d48a8e4624e98acfe6c"
format_stamp: "Formatted at 2026-09-23 20:16:00 on dist-test-slave-vl24"
server_key: "933bcffb2214d2a4156fce4609456058"
server_key_iv: "f3e3acc2ec9da6c2282490fe90a69459"
server_key_version: "encryptionkey@0"
I20260923 20:16:00.318524 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "8ada6626f6f04d48a8e4624e98acfe6c"
format_stamp: "Formatted at 2026-09-23 20:16:00 on dist-test-slave-vl24"
server_key: "933bcffb2214d2a4156fce4609456058"
server_key_iv: "f3e3acc2ec9da6c2282490fe90a69459"
server_key_version: "encryptionkey@0"
I20260923 20:16:00.324144 11928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:16:00.327986 11943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:00.329483 11928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:16:00.329669 11928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "8ada6626f6f04d48a8e4624e98acfe6c"
format_stamp: "Formatted at 2026-09-23 20:16:00 on dist-test-slave-vl24"
server_key: "933bcffb2214d2a4156fce4609456058"
server_key_iv: "f3e3acc2ec9da6c2282490fe90a69459"
server_key_version: "encryptionkey@0"
I20260923 20:16:00.329874 11928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:00.357393 11928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:00.361810 11928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:00.362142 11928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:00.363101 11928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:00.364596 11928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:00.364697 11928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:00.364863 11928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:00.364935 11928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:00.398916 11928 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:40049
I20260923 20:16:00.398970 12055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:40049 every 8 connection(s)
I20260923 20:16:00.400206 11928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:16:00.401798 11789 heartbeater.cc:503] Master 127.5.83.126:42939 was elected leader, sending a full tablet report...
I20260923 20:16:00.403431 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 11928
I20260923 20:16:00.403666 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:16:00.404090 5453 external_mini_cluster.cc:1428] Setting key b911e5d1083ef88e3f45e46c236f4a72
Sep 23 20:16:00 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194560, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:16:00.420378 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:00.403029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:41641 (local address 127.5.83.126:42939)
0923 20:16:00.403253 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:00.403262 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:00.404699 (+ 1437us) server_negotiation.cc:407] Connection header received
0923 20:16:00.406728 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:00.406733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:00.406846 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:00.406925 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:00.408454 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.409295 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.410941 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.411322 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.411414 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:00.414832 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:00.414836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:00.414839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:00.414868 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:00.417401 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:00.418289 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:00.418293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:00.418295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:00.418471 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:00.418951 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:00.418956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:00.418959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:00.419268 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:00.419376 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:00.419990 (+ 614us) server_negotiation.cc:299] Negotiation successful
0923 20:16:00.420143 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 20:16:00.421355 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:00.403642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.67:41641)
0923 20:16:00.404506 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:00.404586 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:00.406449 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:00.407173 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:00.407182 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:00.407657 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:00.408298 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:00.408314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.409412 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.409415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:00.410818 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:00.410830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.411505 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.411509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:00.411693 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:00.412366 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:00.412387 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:00.414637 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:00.417568 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:00.417577 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:00.417592 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:00.418169 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:00.418592 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:00.418597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:00.418600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:00.418800 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:00.419383 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:00.419389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:00.419681 (+ 292us) client_negotiation.cc:769] Sending connection context
0923 20:16:00.419903 (+ 222us) client_negotiation.cc:240] Negotiation successful
0923 20:16:00.420261 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":360,"threads_started":1}
I20260923 20:16:00.422848 12056 heartbeater.cc:347] Connected to a master server at 127.5.83.126:42939
I20260923 20:16:00.423179 12056 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:00.423946 12056 heartbeater.cc:511] Master 127.5.83.126:42939 requested a full tablet report, sending...
I20260923 20:16:00.425329 11600 ts_manager.cc:194] Registered new tserver with Master: 8ada6626f6f04d48a8e4624e98acfe6c (127.5.83.67:40049)
I20260923 20:16:00.426123 11600 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:41641
I20260923 20:16:00.431087 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.83.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.125.keytab.
Entry for principal oryx/127.5.83.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.125.keytab.
I20260923 20:16:00.481956 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.125:45061
--webserver_interface=127.5.83.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.125.keytab
--principal=oryx/127.5.83.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:42939,127.5.83.125:45061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:00.813216 12061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:00.813542 12061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:00.813702 12061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:00.821071 12061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:16:00.821213 12061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:00.821326 12061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:00.821429 12061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:16:00.821506 12061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:16:00.831614 12061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:42939,127.5.83.125:45061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.125.keytab
--principal=oryx/127.5.83.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.125:45061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.83.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
W20260923 20:16:00.832013 12061 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.83.126:42939,127.5.83.125:45061'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 20:16:00.833290 12061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:00.834853 12061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:00.846766 12069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:00.846837 12066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:00.846858 12067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:00.847357 12061 server_base.cc:1034] running on GCE node
I20260923 20:16:00.848223 12061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:00.849751 12061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:00.851048 12061 hybrid_clock.cc:648] HybridClock initialized: now 1790194560850981 us; error 48 us; skew 500 ppm
I20260923 20:16:00.851403 12061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:00 dist-test-slave-vl24 krb5kdc[11555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194560, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:00.854818 12061 init.cc:377] Logged in from keytab as oryx/127.5.83.125@KRBTEST.COM (short username oryx)
I20260923 20:16:00.856315 12061 webserver.cc:460] Webserver started at http://127.5.83.125:44189/ using document root <none> and password file <none>
I20260923 20:16:00.857131 12061 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:00.857266 12061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:00.857667 12061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:00.860719 12061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data/instance:
uuid: "aa3b4ece5be343a683f8719feb5ab14b"
format_stamp: "Formatted at 2026-09-23 20:16:00 on dist-test-slave-vl24"
server_key: "3285f5719642bdf4bcc7744eac4540a6"
server_key_iv: "5588b411c0edf783c3edf066961f3c46"
server_key_version: "encryptionkey@0"
I20260923 20:16:00.861522 12061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/wal/instance:
uuid: "aa3b4ece5be343a683f8719feb5ab14b"
format_stamp: "Formatted at 2026-09-23 20:16:00 on dist-test-slave-vl24"
server_key: "3285f5719642bdf4bcc7744eac4540a6"
server_key_iv: "5588b411c0edf783c3edf066961f3c46"
server_key_version: "encryptionkey@0"
I20260923 20:16:00.867084 12061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:16:00.870962 12076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:00.872459 12061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:16:00.872706 12061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/wal
uuid: "aa3b4ece5be343a683f8719feb5ab14b"
format_stamp: "Formatted at 2026-09-23 20:16:00 on dist-test-slave-vl24"
server_key: "3285f5719642bdf4bcc7744eac4540a6"
server_key_iv: "5588b411c0edf783c3edf066961f3c46"
server_key_version: "encryptionkey@0"
I20260923 20:16:00.872949 12061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:00.912791 12061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:00.913097 11923 heartbeater.cc:503] Master 127.5.83.126:42939 was elected leader, sending a full tablet report...
I20260923 20:16:00.917953 12061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:00.918267 12061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:00.937954 12061 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.125:45061
I20260923 20:16:00.937959 12127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.125:45061 every 8 connection(s)
I20260923 20:16:00.939368 12061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
I20260923 20:16:00.942931 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:00.943989 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 12061
I20260923 20:16:00.952153 12128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:16:00 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.125@KRBTEST.COM
Sep 23 20:16:00 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194560, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.125@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:16:00.969424 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:00.946355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.5.83.125:45061)
0923 20:16:00.947192 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:00.947203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:00.947246 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:16:00.948993 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:00.949016 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:00.949418 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:00.949788 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:00.951619 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.953182 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.954878 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.956267 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.956517 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:00.961792 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:00.961799 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:00.961809 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:00.961847 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:00.965251 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:00.966016 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:00.966025 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:00.966034 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:00.966221 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:00.966701 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:00.966704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:00.966706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:00.967245 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:00.967517 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:00.968014 (+ 497us) server_negotiation.cc:299] Negotiation successful
0923 20:16:00.968350 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":380,"threads_started":1}
I20260923 20:16:00.972831 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:00.954991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.5.83.126:42939)
0923 20:16:00.955504 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:00.955512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:00.956966 (+ 1454us) server_negotiation.cc:407] Connection header received
0923 20:16:00.957095 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:00.957103 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:00.957193 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:00.957292 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:00.958647 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.959822 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.962007 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.962678 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.962834 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:00.966518 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:00.966525 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:00.966530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:00.966576 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:00.969816 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:00.970631 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:00.970634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:00.970637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:00.970834 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:00.971208 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:00.971213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:00.971217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:00.971654 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:00.971813 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:00.972273 (+ 460us) server_negotiation.cc:299] Negotiation successful
0923 20:16:00.972534 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":260,"threads_started":1}
I20260923 20:16:00.972867 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:00.955092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:45850)
0923 20:16:00.956788 (+ 1696us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:00.956853 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:00.956992 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:00.957399 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:00.957408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:00.958213 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:00.958462 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:00.958479 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.959975 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.959980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:00.961847 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:00.961859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.962874 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.962879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:00.963025 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:00.963684 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:00.963719 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:00.966319 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:00.969938 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:00.969947 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:00.969953 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:00.970510 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:00.970951 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:00.970955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:00.970957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:00.971080 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:00.971789 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:00.971794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:00.971933 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 20:16:00.972092 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 20:16:00.972395 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1402,"thread_start_us":403,"threads_started":1}
I20260923 20:16:00.974468 12128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:16:00 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194560, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.125@KRBTEST.COM for oryx/127.5.83.125@KRBTEST.COM
I20260923 20:16:00.986696 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:00.975107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:45061 (local address 127.0.0.1:56504)
0923 20:16:00.975252 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:00.975265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:00.975368 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:00.975669 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:00.975678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:00.976007 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:00.976120 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:00.976127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.977228 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.977234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:00.978144 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:00.978155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.978798 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.978803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:00.979016 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:00.979507 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:00.979522 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:00.981537 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:00.984674 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:00.984677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:00.984680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:00.985015 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:00.985373 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:00.985376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:00.985378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:00.985490 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:00.986021 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:00.986024 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:00.986146 (+ 122us) client_negotiation.cc:769] Sending connection context
0923 20:16:00.986246 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 20:16:00.986456 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":3}
I20260923 20:16:00.986945 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:00.975296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.5.83.125:45061)
0923 20:16:00.975438 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:00.975444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:00.975460 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:16:00.975503 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:00.975505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:00.975548 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:00.975655 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:00.976244 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.977102 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.978279 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:00.978661 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:00.978790 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:00.981710 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:00.981715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:00.981718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:00.981748 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:00.984501 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:00.985136 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:00.985141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:00.985143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:00.985264 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:00.985580 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:00.985583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:00.985585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:00.985905 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:00.986025 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:00.986350 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 20:16:00.986579 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":48}
I20260923 20:16:00.991581 12128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: Bootstrap starting.
I20260923 20:16:00.994627 12128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:00.995623 12128 log.cc:824] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:00.998510 12128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: No bootstrap required, opened a new log
I20260923 20:16:01.006467 12128 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } }
I20260923 20:16:01.006907 12128 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:01.007040 12128 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa3b4ece5be343a683f8719feb5ab14b, State: Initialized, Role: FOLLOWER
I20260923 20:16:01.007771 12128 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } }
I20260923 20:16:01.008881 12135 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } } }
I20260923 20:16:01.009297 12135 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:16:01.010245 12128 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:16:01.020591 12146 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:16:01.020802 12146 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:16:01.024576 12061 master_runner.cc:428] Detected that this master aa3b4ece5be343a683f8719feb5ab14b is joining an existing cluster
I20260923 20:16:01.024689 12061 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.83.125:45061
I20260923 20:16:01.047982 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.029123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.5.83.125:45061)
0923 20:16:01.029341 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.029350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.029982 (+ 632us) server_negotiation.cc:407] Connection header received
0923 20:16:01.030289 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.030294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.030360 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.030433 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.031883 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.033076 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.035159 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.035850 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.035954 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.038935 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.038945 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.038957 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.039005 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.043601 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.044692 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.044701 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.044707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.044856 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.045309 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.045317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.045324 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.046078 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.046248 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.047158 (+ 910us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.047462 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 20:16:01.049623 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.029104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:45061 (local address 127.0.0.1:56520)
0923 20:16:01.029884 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.029906 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.030137 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.030614 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.030619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.031331 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:01.031740 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.031753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.033306 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.033311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.035017 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.035032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.036222 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.036228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.036442 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.037315 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:01.037346 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:01.038695 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:01.043757 (+ 5062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.043766 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.043774 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.044520 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.045016 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.045025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.045031 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.045198 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.046343 (+ 1145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:01.046352 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:01.046568 (+ 216us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.048570 (+ 2002us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.048931 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":4,"thread_start_us":495,"threads_started":1}
I20260923 20:16:01.051826 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.029038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.5.83.126:42939)
0923 20:16:01.029240 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.029248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.034891 (+ 5643us) server_negotiation.cc:407] Connection header received
0923 20:16:01.035182 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.035189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.035303 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.035381 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.037083 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.038184 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.040392 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.041079 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.041190 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.045043 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.045051 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.045056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.045101 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.048669 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.049452 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.049460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.049464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.049621 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.050075 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.050080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.050084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.050460 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.050603 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.051224 (+ 621us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.051478 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:16:01.053154 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.030194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:45854)
0923 20:16:01.034641 (+ 4447us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.034667 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.035017 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.035630 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.035637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.036451 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:01.036887 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.036899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.038392 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.038397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.040216 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.040231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.041468 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.041474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.041645 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.042221 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:01.042248 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:01.044786 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:01.048818 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.048823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.048827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.049303 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.049774 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.049778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.049782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.049934 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.050686 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:01.050690 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:01.050886 (+ 196us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.052054 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.052478 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4284,"mutex_wait_us":80,"thread_start_us":4077,"threads_started":1}
WARNING: no policy specified for oryx/127.5.83.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.124@KRBTEST.COM" created.
Sep 23 20:16:01 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194558, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.126@KRBTEST.COM for oryx/127.5.83.125@KRBTEST.COM
I20260923 20:16:01.079423 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.060182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:45061 (local address 127.0.0.1:56530)
0923 20:16:01.061141 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.061194 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.061421 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.061951 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.061961 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.062871 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:01.063175 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.063187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.064548 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.064554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.066190 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.066205 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.067435 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.067441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.067661 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.068460 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:01.068485 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:01.071629 (+ 3144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:01.076067 (+ 4438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.076085 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.076094 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.076884 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.077461 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.077467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.077471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.077653 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.078378 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:01.078387 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:01.078628 (+ 241us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.078789 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.079044 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":515,"threads_started":1}
I20260923 20:16:01.079607 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.060176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.5.83.125:45061)
0923 20:16:01.060351 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.060357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.061277 (+ 920us) server_negotiation.cc:407] Connection header received
0923 20:16:01.061653 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.061661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.061773 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.061887 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.063349 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.064392 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.066360 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.067241 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.067391 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.071854 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.071862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.071868 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.071911 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.075866 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.077087 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.077094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.077098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.077266 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.077789 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.077794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.077798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.078230 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.078437 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.078942 (+ 505us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.079230 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:16:01.081898 11600 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.83.125:45061: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b"
member_type: NON_VOTER
last_known_addr {
host: "127.5.83.125"
port: 45061
}
attrs {
promote: true
}
}
dest_uuid: "bbcb22e21bc74c539336c817d2b64b05"
cas_config_opid_index: -1
I20260923 20:16:01.083530 11600 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.124.keytab.
Entry for principal oryx/127.5.83.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.124.keytab.
I20260923 20:16:01.088982 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.124:46391
--webserver_interface=127.5.83.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.124.keytab
--principal=oryx/127.5.83.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:42939,127.5.83.125:45061,127.5.83.124:46391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:16:01.089354 12158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER aa3b4ece5be343a683f8719feb5ab14b (127.5.83.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true } } }
I20260923 20:16:01.090198 12103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:01.094458 12103 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Refusing update from remote peer bbcb22e21bc74c539336c817d2b64b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 20:16:01.094439 12158 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.83.125:45061
I20260923 20:16:01.095398 12061 master.cc:537] Master@127.5.83.125:45061 shutting down...
I20260923 20:16:01.095717 12159 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true } } }
I20260923 20:16:01.096127 12159 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:01.097610 12135 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:16:01.097838 12135 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } }
I20260923 20:16:01.097909 12158 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:01.099211 12135 leader_election.cc:288] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939)
I20260923 20:16:01.100832 11610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa3b4ece5be343a683f8719feb5ab14b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbcb22e21bc74c539336c817d2b64b05" is_pre_election: true
I20260923 20:16:01.105432 12080 leader_election.cc:302] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: aa3b4ece5be343a683f8719feb5ab14b; no voters: bbcb22e21bc74c539336c817d2b64b05
I20260923 20:16:01.105978 12135 raft_consensus.cc:2726] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260923 20:16:01.110095 11586 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260923 20:16:01.112100 11586 consensus_peers.cc:487] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 -> Peer aa3b4ece5be343a683f8719feb5ab14b (127.5.83.125:45061): Couldn't send request to peer aa3b4ece5be343a683f8719feb5ab14b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 20:16:01.153357 12061 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:01.153787 12061 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:01.153949 12061 tablet_replica.cc:330] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: stopping tablet replica
I20260923 20:16:01.171198 12061 master.cc:552] Master@127.5.83.125:45061 shutdown complete.
I20260923 20:16:01.171350 12061 master_runner.cc:305] Clearing existing system tablet
I20260923 20:16:01.171864 12061 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:01.171957 12061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:01.175105 12061 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260923 20:16:01.176932 12165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:01.177310 12061 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:01.177719 12061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260923 20:16:01.177845 12061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/wal
uuid: "aa3b4ece5be343a683f8719feb5ab14b"
format_stamp: "Formatted at 2026-09-23 20:16:00 on dist-test-slave-vl24"
server_key: "3285f5719642bdf4bcc7744eac4540a6"
server_key_iv: "5588b411c0edf783c3edf066961f3c46"
server_key_version: "encryptionkey@0"
I20260923 20:16:01.183271 12061 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:01.191090 12061 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 20:16:01.191301 12061 log.cc:1196] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 20:16:01.191804 12061 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: Deleting consensus metadata
I20260923 20:16:01.192126 12061 master_runner.cc:315] Copying system tablet from 127.5.83.126:42939
I20260923 20:16:01.198588 12061 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.83.126:42939
I20260923 20:16:01.215898 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.199680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45858 (local address 127.5.83.126:42939)
0923 20:16:01.199868 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.199874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.201254 (+ 1380us) server_negotiation.cc:407] Connection header received
0923 20:16:01.201448 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.201455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.201558 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.201679 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.202725 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.203898 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.205692 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.206386 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.206528 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.209068 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.209074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.209079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.209117 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.212721 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.213522 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.213528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.213531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.213690 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.214110 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.214117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.214123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.214498 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.214660 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.215151 (+ 491us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.215418 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:16:01.215921 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.199647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:45858)
0923 20:16:01.201094 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.201125 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.201304 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.201732 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.201736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.202264 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:01.202543 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.202556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.204066 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.204071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.205529 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.205545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.206570 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.206576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.206760 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.207495 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:01.207527 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:01.208839 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:01.212857 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.212863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.212867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.213390 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.213791 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.213797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.213801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.213989 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.214629 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:01.214635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:01.214835 (+ 200us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.214990 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.215365 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1259,"thread_start_us":1093,"threads_started":1}
I20260923 20:16:01.217386 11620 tablet_copy_service.cc:140] P bbcb22e21bc74c539336c817d2b64b05: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer aa3b4ece5be343a683f8719feb5ab14b ({username='oryx', principal='oryx/127.5.83.125@KRBTEST.COM'} at 127.0.0.1:45858)
I20260923 20:16:01.217638 11620 tablet_copy_service.cc:161] P bbcb22e21bc74c539336c817d2b64b05: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer aa3b4ece5be343a683f8719feb5ab14b at {username='oryx', principal='oryx/127.5.83.125@KRBTEST.COM'} at 127.0.0.1:45858: session id = aa3b4ece5be343a683f8719feb5ab14b-00000000000000000000000000000000
I20260923 20:16:01.222309 11620 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 20:16:01.224951 12061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 20:16:01.231762 12061 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: tablet copy: Starting download of 0 data blocks...
I20260923 20:16:01.232061 12061 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: tablet copy: Starting download of 1 WAL segments...
I20260923 20:16:01.235683 12061 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 20:16:01.240150 11620 tablet_copy_service.cc:342] P bbcb22e21bc74c539336c817d2b64b05: Request end of tablet copy session aa3b4ece5be343a683f8719feb5ab14b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.83.125@KRBTEST.COM'} at 127.0.0.1:45858
I20260923 20:16:01.240356 11620 tablet_copy_service.cc:433] P bbcb22e21bc74c539336c817d2b64b05: ending tablet copy session aa3b4ece5be343a683f8719feb5ab14b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer aa3b4ece5be343a683f8719feb5ab14b
I20260923 20:16:01.253314 12061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:01.255831 12172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:01.257169 12173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:01.263788 12175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:01.269869 12061 server_base.cc:1034] running on GCE node
I20260923 20:16:01.270395 12061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:01.271220 12061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:01.272462 12061 hybrid_clock.cc:648] HybridClock initialized: now 1790194561272402 us; error 32 us; skew 500 ppm
I20260923 20:16:01.272718 12061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:01 dist-test-slave-vl24 krb5kdc[11555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194561, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:01.275097 12061 init.cc:377] Logged in from keytab as oryx/127.5.83.125@KRBTEST.COM (short username oryx)
I20260923 20:16:01.276471 12061 webserver.cc:460] Webserver started at http://127.5.83.125:45871/ using document root <none> and password file <none>
I20260923 20:16:01.276875 12061 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:01.277009 12061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:01.280144 12061 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260923 20:16:01.282204 12182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:01.282649 12061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 20:16:01.282778 12061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/wal
uuid: "aa3b4ece5be343a683f8719feb5ab14b"
format_stamp: "Formatted at 2026-09-23 20:16:00 on dist-test-slave-vl24"
server_key: "3285f5719642bdf4bcc7744eac4540a6"
server_key_iv: "5588b411c0edf783c3edf066961f3c46"
server_key_version: "encryptionkey@0"
I20260923 20:16:01.282950 12061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:01.305881 12061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:01.310101 12061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:01.344201 12061 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.125:45061
I20260923 20:16:01.344233 12233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.125:45061 every 8 connection(s)
I20260923 20:16:01.345635 12061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
I20260923 20:16:01.351761 12234 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:16:01.356901 12234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: Bootstrap starting.
I20260923 20:16:01.376325 12234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:16:01.377024 12234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: Bootstrap complete.
I20260923 20:16:01.378242 12234 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true } }
I20260923 20:16:01.378597 12234 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa3b4ece5be343a683f8719feb5ab14b, State: Initialized, Role: LEARNER
I20260923 20:16:01.378863 12234 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true } }
I20260923 20:16:01.379755 12237 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true } } }
I20260923 20:16:01.380028 12234 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:16:01.380107 12237 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: This master's current role is: LEARNER
I20260923 20:16:01.401051 12248 catalog_manager.cc:1259] Loaded cluster ID: 8028ca1dcc2a4dd2a2f380dd6787ff86
I20260923 20:16:01.401278 12248 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: loading cluster ID for follower catalog manager: success
I20260923 20:16:01.405185 12248 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: acquiring CA information for follower catalog manager: success
I20260923 20:16:01.408052 12248 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:16:01.428493 12056 heartbeater.cc:503] Master 127.5.83.126:42939 was elected leader, sending a full tablet report...
W20260923 20:16:01.502948 12162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:01.503216 12162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:01.503302 12162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:01.511353 12162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:16:01.511468 12162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:01.511518 12162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:01.511575 12162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:16:01.511618 12162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:16:01.522022 12162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:42939,127.5.83.125:45061,127.5.83.124:46391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.124.keytab
--principal=oryx/127.5.83.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.124:46391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.83.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:01.523440 12162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:01.524747 12162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:01.537214 12254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:01.537263 12253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:01.537179 12256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:01.538443 12162 server_base.cc:1034] running on GCE node
I20260923 20:16:01.539294 12162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:01.540792 12162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:01.542023 12162 hybrid_clock.cc:648] HybridClock initialized: now 1790194561541975 us; error 34 us; skew 500 ppm
I20260923 20:16:01.542338 12162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:01 dist-test-slave-vl24 krb5kdc[11555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194561, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:01.545722 12162 init.cc:377] Logged in from keytab as oryx/127.5.83.124@KRBTEST.COM (short username oryx)
I20260923 20:16:01.547384 12162 webserver.cc:460] Webserver started at http://127.5.83.124:43585/ using document root <none> and password file <none>
I20260923 20:16:01.548103 12162 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:01.548272 12162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:01.548636 12162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:01.551653 12162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data/instance:
uuid: "2316a8bbd88e4697bcc22dc8943a72d2"
format_stamp: "Formatted at 2026-09-23 20:16:01 on dist-test-slave-vl24"
server_key: "0bb9636259fbcd47452c2bd806c6bedf"
server_key_iv: "694c90fb577c15ce5fc23f1029342290"
server_key_version: "encryptionkey@0"
I20260923 20:16:01.552376 12162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/wal/instance:
uuid: "2316a8bbd88e4697bcc22dc8943a72d2"
format_stamp: "Formatted at 2026-09-23 20:16:01 on dist-test-slave-vl24"
server_key: "0bb9636259fbcd47452c2bd806c6bedf"
server_key_iv: "694c90fb577c15ce5fc23f1029342290"
server_key_version: "encryptionkey@0"
I20260923 20:16:01.557677 12162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260923 20:16:01.561343 12263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:01.562990 12162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:16:01.563128 12162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/wal
uuid: "2316a8bbd88e4697bcc22dc8943a72d2"
format_stamp: "Formatted at 2026-09-23 20:16:01 on dist-test-slave-vl24"
server_key: "0bb9636259fbcd47452c2bd806c6bedf"
server_key_iv: "694c90fb577c15ce5fc23f1029342290"
server_key_version: "encryptionkey@0"
I20260923 20:16:01.563289 12162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:01.573645 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.566511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.5.83.125:45061)
0923 20:16:01.567323 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.567327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.567342 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:16:01.567399 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.567402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.567484 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.567598 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:01.568119 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.569640 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.571811 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.572736 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.572841 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.573007 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.573297 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.573393 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":571,"threads_started":1}
I20260923 20:16:01.573647 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.566296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:45061 (local address 127.0.0.1:56538)
0923 20:16:01.566497 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.566518 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.566682 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.567688 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.567693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.567725 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:01.567945 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.567957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.569795 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.569800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.571672 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.571687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.572895 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.572901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.573113 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.573169 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.573260 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.573389 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 20:16:01.574792 12209 raft_consensus.cc:1203] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260923 20:16:01.575309 12237 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbcb22e21bc74c539336c817d2b64b05. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true } } }
I20260923 20:16:01.575598 12237 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: This master's current role is: LEARNER
I20260923 20:16:01.590942 12162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:01.595309 12162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:01.595595 12162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:01.615401 12162 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.124:46391
I20260923 20:16:01.615486 12315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.124:46391 every 8 connection(s)
I20260923 20:16:01.616860 12162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
I20260923 20:16:01.619906 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 12162
I20260923 20:16:01.620841 12316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:01.630398 12316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:16:01 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.124@KRBTEST.COM
Sep 23 20:16:01 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194561, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.124@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:16:01.646139 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.622115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52082 (local address 127.5.83.124:46391)
0923 20:16:01.623081 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.623093 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.623131 (+ 38us) server_negotiation.cc:407] Connection header received
0923 20:16:01.625456 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.625487 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.625860 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.626196 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.628104 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.629453 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.631229 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.631989 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.632119 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.638065 (+ 5946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.638073 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.638083 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.638120 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.641339 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.642197 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.642204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.642210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.642436 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.642824 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.642835 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.642838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.643465 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.643779 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.644252 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.644740 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":542,"threads_started":1}
I20260923 20:16:01.649331 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.633465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.5.83.126:42939)
0923 20:16:01.633691 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.633700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.634447 (+ 747us) server_negotiation.cc:407] Connection header received
0923 20:16:01.634695 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.634700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.634762 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.634842 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.636281 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.637516 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.639112 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.639494 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.639644 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.643151 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.643157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.643162 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.643201 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.646568 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.647246 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.647250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.647254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.647407 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.647850 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.647853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.647855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.648133 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.648241 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.648753 (+ 512us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.649001 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:16:01.649436 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.633393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:45862)
0923 20:16:01.634262 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.634344 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.634551 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.634946 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.634954 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.635778 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:01.636119 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.636129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.637686 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.637689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.638947 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.638958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.639684 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.639691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.639915 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.640476 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:01.640493 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:01.642967 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:01.646694 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.646704 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.646709 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.647128 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.647514 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.647520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.647527 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.647712 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.648256 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:01.648262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:01.648453 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.648681 (+ 228us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.649008 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":421,"threads_started":1}
I20260923 20:16:01.650718 12316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:16:01 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194561, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.124@KRBTEST.COM for oryx/127.5.83.125@KRBTEST.COM
I20260923 20:16:01.662524 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.651355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:45061 (local address 127.0.0.1:56542)
0923 20:16:01.651562 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.651577 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.651652 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.651896 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.651898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.652172 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:01.652288 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.652295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.653313 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.653316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.654208 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.654219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.654838 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.654842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.654954 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.655623 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:01.655640 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:01.657558 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:01.660446 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.660449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.660452 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.660898 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.661238 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.661241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.661243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.661377 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.661878 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:01.661881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:01.661993 (+ 112us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.662112 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.662301 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 20:16:01.662591 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.651444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.5.83.125:45061)
0923 20:16:01.651618 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.651622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.651638 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:16:01.651731 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.651735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.651790 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.651864 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.652417 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.653196 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.654322 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.654717 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.654806 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.657731 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.657736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.657740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.657770 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.660324 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.660995 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.660999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.661001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.661143 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.661504 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.661508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.661510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.661776 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.661894 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.662191 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.662359 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:16:01.663556 12316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:16:01 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194561, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.124@KRBTEST.COM for oryx/127.5.83.124@KRBTEST.COM
I20260923 20:16:01.675873 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.664483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.5.83.124:46391)
0923 20:16:01.664638 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.664643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.664660 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:16:01.664733 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.664738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.664799 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.664894 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.665518 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.666194 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.667417 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.667902 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.668044 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.671021 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.671027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.671031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.671068 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.673400 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.673995 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.674001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.674006 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.674125 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.674515 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.674520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.674523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.674765 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.674908 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.675240 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.675559 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":69}
I20260923 20:16:01.675879 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.664301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:46391 (local address 127.0.0.1:52094)
0923 20:16:01.664482 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.664497 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.664628 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.664909 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.664912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.665253 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:01.665382 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.665389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.666320 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.666325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.667275 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.667286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.668055 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.668060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.668304 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.668912 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:01.668930 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:01.670848 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:01.673524 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.673531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.673537 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.673866 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.674244 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.674249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.674252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.674399 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.674873 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:01.674878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:01.675031 (+ 153us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.675165 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.675429 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":3}
I20260923 20:16:01.680795 12316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: Bootstrap starting.
I20260923 20:16:01.683920 12316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:01.685062 12316 log.cc:824] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:01.687656 12316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: No bootstrap required, opened a new log
I20260923 20:16:01.694849 12316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } }
I20260923 20:16:01.695145 12316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:01.695276 12316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2316a8bbd88e4697bcc22dc8943a72d2, State: Initialized, Role: FOLLOWER
I20260923 20:16:01.695843 12316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } }
I20260923 20:16:01.696759 12322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } } }
I20260923 20:16:01.697121 12322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:16:01.697937 12316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:16:01.707335 12333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:16:01.707484 12333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:16:01.712167 12162 master_runner.cc:428] Detected that this master 2316a8bbd88e4697bcc22dc8943a72d2 is joining an existing cluster
I20260923 20:16:01.712343 12162 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.83.124:46391
I20260923 20:16:01.735520 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.716453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.5.83.125:45061)
0923 20:16:01.716703 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.716708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.717829 (+ 1121us) server_negotiation.cc:407] Connection header received
0923 20:16:01.718056 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.718067 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.718130 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.718231 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.719373 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.720137 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.722873 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.723759 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.723873 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.728131 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.728139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.728146 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.728190 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.731876 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.732719 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.732727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.732732 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.732884 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.733510 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.733515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.733520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.733937 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.734085 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.734835 (+ 750us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.735094 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260923 20:16:01.736748 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.717036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:45061 (local address 127.0.0.1:56550)
0923 20:16:01.717715 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.717743 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.717899 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.718253 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.718257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.718938 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:01.719226 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.719235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.720272 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.720281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.722605 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.722673 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.725564 (+ 2891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.725576 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.725804 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.726741 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:01.726774 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:01.727911 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:01.732052 (+ 4141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.732057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.732061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.732568 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.733065 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.733069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.733073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.733312 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.734164 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:01.734170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:01.734401 (+ 231us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.735745 (+ 1344us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.736140 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"spinlock_wait_cycles":28032,"thread_start_us":413,"threads_started":1}
I20260923 20:16:01.742532 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.716426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.5.83.126:42939)
0923 20:16:01.716668 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.716674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.724810 (+ 8136us) server_negotiation.cc:407] Connection header received
0923 20:16:01.725159 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.725166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.725251 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.725323 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.727062 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.728212 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.730404 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.731074 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.731173 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.734747 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.734753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.734758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.734798 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.739055 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.740171 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.740179 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.740185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.740349 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.740840 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.740846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.740850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.741199 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.741363 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.741890 (+ 527us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.742157 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:16:01.742552 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.720868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:45872)
0923 20:16:01.724660 (+ 3792us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.724708 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.724930 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.725556 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.725561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.726264 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:01.726866 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.726880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.728425 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.728431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.730229 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.730243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.731750 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.731760 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.731953 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.733031 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:01.733057 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:01.734516 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:01.739365 (+ 4849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.739370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.739376 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.740016 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.740500 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.740507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.740511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.740701 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.741318 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:01.741322 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:01.741528 (+ 206us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.741731 (+ 203us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.742081 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3590,"mutex_wait_us":55,"thread_start_us":3370,"threads_started":1}
I20260923 20:16:01.742601 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.716627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.5.83.124:46391)
0923 20:16:01.716803 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.716807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.724218 (+ 7411us) server_negotiation.cc:407] Connection header received
0923 20:16:01.724697 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.724705 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.724785 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.724858 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.726371 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.727801 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.729949 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.731925 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.732076 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.734787 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.734795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.734799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.734827 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.738187 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.740374 (+ 2187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.740379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.740382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.740515 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.741065 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.741070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.741074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.741498 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.741670 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.742050 (+ 380us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.742262 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":60}
I20260923 20:16:01.743675 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.717191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:46391 (local address 127.0.0.1:52110)
0923 20:16:01.724116 (+ 6925us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.724142 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.724521 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.725063 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.725067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.725843 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:01.726206 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.726219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.728034 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.728042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.729785 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.729800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.732223 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.732230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.732390 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.733239 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:01.733262 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:01.734623 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:01.738385 (+ 3762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.738390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.738395 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.740195 (+ 1800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.740677 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.740681 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.740684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.740875 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.741628 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:01.741633 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:01.741797 (+ 164us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.742866 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.743120 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6752,"thread_start_us":6534,"threads_started":1}
I20260923 20:16:01.752398 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 11661
Sep 23 20:16:01 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194558, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.126@KRBTEST.COM for oryx/127.5.83.124@KRBTEST.COM
Sep 23 20:16:01 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.65@KRBTEST.COM
I20260923 20:16:01.771682 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.753575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.5.83.65:40173)
0923 20:16:01.754213 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.754223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.754254 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:16:01.754330 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.754340 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.754613 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.754793 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.756364 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.757507 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.759412 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.760317 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.761244 (+ 927us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.765409 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.765420 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.765435 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.765466 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.768854 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.769546 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.769554 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.769558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.769688 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.770061 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.770067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.770071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.770401 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.770707 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.771031 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.771292 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":572,"threads_started":1}
I20260923 20:16:01.773001 11784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:01.773685 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.748428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:46391 (local address 127.0.0.1:52120)
0923 20:16:01.748667 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.748687 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.748787 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.749190 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.749195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.749686 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:01.749879 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.749891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.751075 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.751081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.752472 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.752487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.753666 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.753672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.753859 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.754489 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:01.754506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:01.757380 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:01.761151 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.761158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.761162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.761664 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.762104 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.762109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.762113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.762269 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.772608 (+ 10339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:01.772614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:01.772807 (+ 193us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.772990 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.773296 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 20:16:01.774024 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.748567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52120 (local address 127.5.83.124:46391)
0923 20:16:01.748709 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.748713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.748727 (+ 14us) server_negotiation.cc:407] Connection header received
0923 20:16:01.748926 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.748930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.749047 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.749129 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.750078 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.750919 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.752681 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.753487 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.753620 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.757585 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.757593 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.757601 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.757648 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.761020 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.761812 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.761819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.761823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.761976 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.762407 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.762415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.762419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.762798 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.763004 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.773175 (+ 10171us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.773514 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 20:16:01.776208 11600 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.83.124:46391: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2"
member_type: NON_VOTER
last_known_addr {
host: "127.5.83.124"
port: 46391
}
attrs {
promote: true
}
}
dest_uuid: "bbcb22e21bc74c539336c817d2b64b05"
cas_config_opid_index: 5
I20260923 20:16:01.777649 11600 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } }
I20260923 20:16:01.780195 12159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2316a8bbd88e4697bcc22dc8943a72d2 (127.5.83.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } } }
I20260923 20:16:01.784526 12291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:01.791375 12159 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.83.124:46391
I20260923 20:16:01.793329 12291 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 FOLLOWER]: Refusing update from remote peer bbcb22e21bc74c539336c817d2b64b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 20:16:01.794256 12162 master.cc:537] Master@127.5.83.124:46391 shutting down...
I20260923 20:16:01.802636 12209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 LEARNER]: Refusing update from remote peer bbcb22e21bc74c539336c817d2b64b05: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 20:16:01.809000 12347 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:01.809438 12345 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } } }
I20260923 20:16:01.809799 12159 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 20:16:01.810015 12345 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:01.819728 12209 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2316a8bbd88e4697bcc22dc8943a72d2 (127.5.83.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } } }
I20260923 20:16:01.823403 12237 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: NON_VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: true } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } } }
I20260923 20:16:01.823712 12237 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: This master's current role is: LEARNER
I20260923 20:16:01.824237 12344 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05: attempting to promote NON_VOTER aa3b4ece5be343a683f8719feb5ab14b to VOTER
I20260923 20:16:01.825112 12344 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } }
I20260923 20:16:01.827366 12209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 LEARNER]: Refusing update from remote peer bbcb22e21bc74c539336c817d2b64b05: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 20:16:01.827944 12159 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 20:16:01.830430 12348 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, aa3b4ece5be343a683f8719feb5ab14b (127.5.83.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } } }
I20260923 20:16:01.835139 12162 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:01.835638 12162 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:01.835588 12344 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } } }
I20260923 20:16:01.835834 12162 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: stopping tablet replica
I20260923 20:16:01.835618 12348 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } } }
I20260923 20:16:01.835862 12344 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:01.836004 12348 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:01.838644 12209 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, aa3b4ece5be343a683f8719feb5ab14b (127.5.83.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } } }
I20260923 20:16:01.842348 12162 master.cc:552] Master@127.5.83.124:46391 shutdown complete.
I20260923 20:16:01.842422 12162 master_runner.cc:305] Clearing existing system tablet
I20260923 20:16:01.842833 12162 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:01.842953 12162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:01.846525 12237 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } } }
I20260923 20:16:01.846766 12237 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:16:01.853333 12162 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.007s
I20260923 20:16:01.855108 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.854627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:46391 (local address 127.0.0.1:52124)
0923 20:16:01.854815 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.854939 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260923 20:16:01.855325 11584 consensus_peers.cc:487] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 -> Peer 2316a8bbd88e4697bcc22dc8943a72d2 (127.5.83.124:46391): Couldn't send request to peer 2316a8bbd88e4697bcc22dc8943a72d2. Status: Network error: Client connection negotiation failed: client connection to 127.5.83.124:46391: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 20:16:01.857340 12353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:01.857713 12162 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:01.858132 12162 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260923 20:16:01.858276 12162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/wal
uuid: "2316a8bbd88e4697bcc22dc8943a72d2"
format_stamp: "Formatted at 2026-09-23 20:16:01 on dist-test-slave-vl24"
server_key: "0bb9636259fbcd47452c2bd806c6bedf"
server_key_iv: "694c90fb577c15ce5fc23f1029342290"
server_key_version: "encryptionkey@0"
I20260923 20:16:01.863543 12162 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:01.870639 12162 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 20:16:01.870873 12162 log.cc:1196] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 20:16:01.871470 12162 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: Deleting consensus metadata
I20260923 20:16:01.871778 12162 master_runner.cc:315] Copying system tablet from 127.5.83.126:42939
I20260923 20:16:01.874413 12162 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: tablet copy: Beginning tablet copy session from remote peer at address 127.5.83.126:42939
I20260923 20:16:01.890681 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.875326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.5.83.126:42939)
0923 20:16:01.875619 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:01.875626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:01.875907 (+ 281us) server_negotiation.cc:407] Connection header received
0923 20:16:01.876085 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:01.876091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:01.876174 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:01.876281 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:01.877366 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.878488 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.880428 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.881223 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.881338 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.884015 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:01.884022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:01.884027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:01.884072 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:01.887707 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.888458 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.888465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.888469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.888680 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:01.889121 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:01.889126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:01.889129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:01.889466 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:01.889617 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:01.890071 (+ 454us) server_negotiation.cc:299] Negotiation successful
0923 20:16:01.890315 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260923 20:16:01.890735 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:01.875267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:45882)
0923 20:16:01.875791 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:01.875811 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:01.875955 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:01.876327 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:01.876332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:01.876922 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:01.877205 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.877218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.878656 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.878661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.880269 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:01.880287 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:01.881612 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:01.881618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:01.881786 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:01.882506 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:01.882533 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:01.883753 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:01.887853 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.887859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.887864 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.888332 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.888792 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:01.888797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:01.888801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:01.889006 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:01.889614 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:01.889619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:01.889809 (+ 190us) client_negotiation.cc:769] Sending connection context
0923 20:16:01.889951 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 20:16:01.890281 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":357,"threads_started":1}
I20260923 20:16:01.891638 11620 tablet_copy_service.cc:140] P bbcb22e21bc74c539336c817d2b64b05: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2316a8bbd88e4697bcc22dc8943a72d2 ({username='oryx', principal='oryx/127.5.83.124@KRBTEST.COM'} at 127.0.0.1:45882)
I20260923 20:16:01.891816 11620 tablet_copy_service.cc:161] P bbcb22e21bc74c539336c817d2b64b05: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2316a8bbd88e4697bcc22dc8943a72d2 at {username='oryx', principal='oryx/127.5.83.124@KRBTEST.COM'} at 127.0.0.1:45882: session id = 2316a8bbd88e4697bcc22dc8943a72d2-00000000000000000000000000000000
I20260923 20:16:01.895533 11620 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 20:16:01.897678 12162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 20:16:01.904372 12162 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: tablet copy: Starting download of 0 data blocks...
I20260923 20:16:01.904718 12162 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: tablet copy: Starting download of 1 WAL segments...
I20260923 20:16:01.907366 12162 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 20:16:01.911713 11620 tablet_copy_service.cc:342] P bbcb22e21bc74c539336c817d2b64b05: Request end of tablet copy session 2316a8bbd88e4697bcc22dc8943a72d2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.83.124@KRBTEST.COM'} at 127.0.0.1:45882
I20260923 20:16:01.911866 11620 tablet_copy_service.cc:433] P bbcb22e21bc74c539336c817d2b64b05: ending tablet copy session 2316a8bbd88e4697bcc22dc8943a72d2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2316a8bbd88e4697bcc22dc8943a72d2
I20260923 20:16:01.923717 12162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:01.926412 12360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:01.926666 12361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:01.930151 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 11661
I20260923 20:16:01.948158 12162 server_base.cc:1034] running on GCE node
W20260923 20:16:01.949402 12363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:01.949939 12162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:01.950659 12162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:01.951886 12162 hybrid_clock.cc:648] HybridClock initialized: now 1790194561951840 us; error 35 us; skew 500 ppm
I20260923 20:16:01.952136 12162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:01 dist-test-slave-vl24 krb5kdc[11555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194561, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:01.954322 12162 init.cc:377] Logged in from keytab as oryx/127.5.83.124@KRBTEST.COM (short username oryx)
I20260923 20:16:01.954533 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:40173
--local_ip_for_outbound_sockets=127.5.83.65
--tserver_master_addrs=127.5.83.126:42939
--webserver_port=39417
--webserver_interface=127.5.83.65
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab
--principal=oryx/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.83.126:42939,127.5.83.125:45061
--tserver_master_addrs=127.5.83.126:42939,127.5.83.125:45061,127.5.83.124:46391 with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:16:01.956046 12162 webserver.cc:460] Webserver started at http://127.5.83.124:39999/ using document root <none> and password file <none>
I20260923 20:16:01.956372 12162 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:01.956488 12162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:01.959781 12162 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260923 20:16:01.961854 12371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:01.962282 12162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 20:16:01.962426 12162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/wal
uuid: "2316a8bbd88e4697bcc22dc8943a72d2"
format_stamp: "Formatted at 2026-09-23 20:16:01 on dist-test-slave-vl24"
server_key: "0bb9636259fbcd47452c2bd806c6bedf"
server_key_iv: "694c90fb577c15ce5fc23f1029342290"
server_key_version: "encryptionkey@0"
I20260923 20:16:01.962704 12162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:01.973788 12162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:01.978081 12162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:01.999032 12162 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.124:46391
I20260923 20:16:01.999087 12422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.124:46391 every 8 connection(s)
I20260923 20:16:02.000183 12162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
I20260923 20:16:02.005918 12423 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:16:02.010152 12423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: Bootstrap starting.
I20260923 20:16:02.029227 12423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:16:02.029757 12423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: Bootstrap complete.
I20260923 20:16:02.030795 12423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } }
I20260923 20:16:02.031141 12423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2316a8bbd88e4697bcc22dc8943a72d2, State: Initialized, Role: LEARNER
I20260923 20:16:02.031400 12423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } }
I20260923 20:16:02.032204 12426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } } }
I20260923 20:16:02.032473 12423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:16:02.032675 12426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: This master's current role is: LEARNER
I20260923 20:16:02.045853 12437 catalog_manager.cc:1259] Loaded cluster ID: 8028ca1dcc2a4dd2a2f380dd6787ff86
I20260923 20:16:02.045962 12437 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: loading cluster ID for follower catalog manager: success
I20260923 20:16:02.049381 12437 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: acquiring CA information for follower catalog manager: success
I20260923 20:16:02.051735 12437 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:16:02.260320 12438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:02.253538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.5.83.124:46391)
0923 20:16:02.254093 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:02.254097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:02.254113 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:16:02.254173 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:02.254176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:02.254243 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:02.254344 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:02.254838 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.255996 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.258510 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.259343 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.259450 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:02.259641 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:02.259964 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 20:16:02.260064 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":310,"threads_started":1}
I20260923 20:16:02.260353 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:02.253340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:46391 (local address 127.0.0.1:52138)
0923 20:16:02.253560 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:02.253585 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:02.253732 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:02.254408 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:02.254415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:02.254461 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:02.254667 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:02.254681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.256169 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.256176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:02.258306 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:02.258325 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.259538 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.259547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:02.259742 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:02.259783 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 20:16:02.259900 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 20:16:02.260061 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 20:16:02.261513 12398 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 20:16:02.262171 12426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbcb22e21bc74c539336c817d2b64b05. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } } }
I20260923 20:16:02.262502 12426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: This master's current role is: LEARNER
W20260923 20:16:02.317363 12370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:02.317656 12370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:02.317839 12370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:02.325470 12370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:02.325578 12370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:02.325760 12370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:16:02.335911 12370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab
--principal=oryx/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:40173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=39417
--tserver_master_addrs=127.5.83.126:42939,127.5.83.125:45061,127.5.83.124:46391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:02.337705 12370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:02.339046 12370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:02.353902 12443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:02.354023 12444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:02.355068 12446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:02.355603 12370 server_base.cc:1034] running on GCE node
I20260923 20:16:02.356297 12370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:02.357529 12370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:02.358768 12370 hybrid_clock.cc:648] HybridClock initialized: now 1790194562358718 us; error 40 us; skew 500 ppm
I20260923 20:16:02.359120 12370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:02 dist-test-slave-vl24 krb5kdc[11555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194562, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:02.363135 12370 init.cc:377] Logged in from keytab as oryx/127.5.83.65@KRBTEST.COM (short username oryx)
I20260923 20:16:02.364734 12370 webserver.cc:460] Webserver started at http://127.5.83.65:39417/ using document root <none> and password file <none>
I20260923 20:16:02.365437 12370 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:02.365579 12370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:02.371001 12370 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.002s
I20260923 20:16:02.373890 12453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:02.375273 12370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:16:02.375450 12370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "8eb6ee177604473fa83b0371abc96e51"
format_stamp: "Formatted at 2026-09-23 20:15:59 on dist-test-slave-vl24"
server_key: "68177ef81ab7876d87c5416b5de73720"
server_key_iv: "aedd558d9b885edf3c0098de4f243e62"
server_key_version: "encryptionkey@0"
I20260923 20:16:02.376569 12370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:02.418390 12370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:02.422910 12370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:02.423213 12370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:02.424244 12370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:02.425683 12370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:02.425796 12370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:02.425931 12370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:02.426033 12370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:02.461714 12370 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:40173
I20260923 20:16:02.461719 12565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:40173 every 8 connection(s)
I20260923 20:16:02.463145 12370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:16:02.469275 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 12370
I20260923 20:16:02.469656 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 11795
Sep 23 20:16:02 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.66@KRBTEST.COM
Sep 23 20:16:02 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194562, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.124@KRBTEST.COM
Sep 23 20:16:02 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194562, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.125@KRBTEST.COM
I20260923 20:16:02.503328 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:02.470823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.5.83.66:45889)
0923 20:16:02.482479 (+ 11656us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:02.482488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:02.482515 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:16:02.482576 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:02.482584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:02.482834 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:02.483034 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:02.484946 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.486125 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.487472 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.488069 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.488215 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:02.494206 (+ 5991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:02.494219 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:02.494234 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:02.494293 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:02.498585 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:02.499503 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:02.499511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:02.499515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:02.499677 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:02.500123 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:02.500129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:02.500133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:02.500597 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:02.500869 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:02.502753 (+ 1884us) server_negotiation.cc:299] Negotiation successful
0923 20:16:02.503015 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11475,"thread_start_us":11323,"threads_started":1}
I20260923 20:16:02.504769 11918 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 23 20:16:02 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194562, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:16:02.509943 12438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:02.467833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:56149 (local address 127.5.83.124:46391)
0923 20:16:02.468068 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:02.468077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:02.485410 (+ 17333us) server_negotiation.cc:407] Connection header received
0923 20:16:02.488983 (+ 3573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:02.488991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:02.489084 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:02.489197 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:02.491773 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.492904 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.495817 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.496592 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.496710 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:02.501707 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:02.501715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:02.501722 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:02.501765 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:02.506139 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:02.507179 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:02.507187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:02.507193 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:02.507372 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:02.507895 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:02.507903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:02.507908 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:02.508281 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:02.508412 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:02.509396 (+ 984us) server_negotiation.cc:299] Negotiation successful
0923 20:16:02.509620 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:16:02.511032 12576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:02.466973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:46391 (local address 127.5.83.65:56149)
0923 20:16:02.485228 (+ 18255us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:02.485277 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:02.488558 (+ 3281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:02.489661 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:02.489674 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:02.490499 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:02.491507 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:02.491535 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.493133 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.493139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:02.495630 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:02.495648 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.497070 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.497080 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:02.497356 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:02.498071 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:02.498105 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:02.501477 (+ 3372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:02.506305 (+ 4828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:02.506316 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:02.506334 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:02.507008 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:02.507519 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:02.507523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:02.507527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:02.507765 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:02.508515 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:02.508542 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:02.508956 (+ 414us) client_negotiation.cc:769] Sending connection context
0923 20:16:02.509322 (+ 366us) client_negotiation.cc:240] Negotiation successful
0923 20:16:02.509786 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17942,"mutex_wait_us":31,"thread_start_us":30860,"threads_started":1}
I20260923 20:16:02.513935 12567 heartbeater.cc:347] Connected to a master server at 127.5.83.124:46391
I20260923 20:16:02.514220 12567 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:02.515146 12567 heartbeater.cc:511] Master 127.5.83.124:46391 requested a full tablet report, sending...
I20260923 20:16:02.515417 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:02.467722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:49907 (local address 127.5.83.126:42939)
0923 20:16:02.478787 (+ 11065us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:02.478796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:02.498434 (+ 19638us) server_negotiation.cc:407] Connection header received
0923 20:16:02.498624 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:02.498630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:02.498716 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:02.498791 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:02.499964 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.501133 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.503853 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.504480 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.504649 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:02.507757 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:02.507766 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:02.507773 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:02.507819 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:02.512349 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:02.513249 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:02.513257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:02.513263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:02.513406 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:02.513856 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:02.513861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:02.513865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:02.514193 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:02.514309 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:02.514854 (+ 545us) server_negotiation.cc:299] Negotiation successful
0923 20:16:02.515077 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10878,"thread_start_us":11077,"threads_started":1}
I20260923 20:16:02.516604 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:02.468358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.65:49907)
0923 20:16:02.498286 (+ 29928us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:02.498304 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:02.498454 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:02.499013 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:02.499016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:02.499514 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:02.499805 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:02.499816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.502074 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.502079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:02.503696 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:02.503712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.504679 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.504685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:02.504853 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:02.505446 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:02.505471 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:02.507546 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:02.512506 (+ 4960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:02.512509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:02.512511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:02.513047 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:02.513564 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:02.513569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:02.513573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:02.513729 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:02.514398 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:02.514403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:02.514549 (+ 146us) client_negotiation.cc:769] Sending connection context
0923 20:16:02.515663 (+ 1114us) client_negotiation.cc:240] Negotiation successful
0923 20:16:02.515968 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29802,"thread_start_us":20900,"threads_started":1}
I20260923 20:16:02.517606 12566 heartbeater.cc:347] Connected to a master server at 127.5.83.126:42939
I20260923 20:16:02.517750 12566 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:02.518162 12566 heartbeater.cc:511] Master 127.5.83.126:42939 requested a full tablet report, sending...
I20260923 20:16:02.518832 12388 ts_manager.cc:194] Registered new tserver with Master: 8eb6ee177604473fa83b0371abc96e51 (127.5.83.65:40173)
I20260923 20:16:02.519397 11600 ts_manager.cc:194] Re-registered known tserver with Master: 8eb6ee177604473fa83b0371abc96e51 (127.5.83.65:40173)
I20260923 20:16:02.519912 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:02.466973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:45061 (local address 127.5.83.65:50957)
0923 20:16:02.489548 (+ 22575us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:02.489573 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:02.489720 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:02.490434 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:02.490442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:02.491297 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:02.491769 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:02.491784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.493864 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.493873 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:02.496313 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:02.496332 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.500032 (+ 3700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.500040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:02.500206 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:02.500808 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:02.500828 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:02.511051 (+ 10223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:02.516019 (+ 4968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:02.516026 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:02.516031 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:02.516633 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:02.517085 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:02.517089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:02.517092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:02.517232 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:02.518916 (+ 1684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:02.518924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:02.519073 (+ 149us) client_negotiation.cc:769] Sending connection context
0923 20:16:02.519195 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 20:16:02.519467 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22403,"mutex_wait_us":342,"thread_start_us":16658,"threads_started":1}
I20260923 20:16:02.520601 11600 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:49907
I20260923 20:16:02.520888 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:02.466187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:50957 (local address 127.5.83.125:45061)
0923 20:16:02.482846 (+ 16659us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:02.482855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:02.489848 (+ 6993us) server_negotiation.cc:407] Connection header received
0923 20:16:02.489933 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:02.489942 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:02.490074 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:02.490193 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:02.492007 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.493654 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.496554 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:02.497461 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:02.497602 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:02.511727 (+ 14125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:02.511735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:02.511739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:02.511785 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:02.515843 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:02.516791 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:02.516797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:02.516801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:02.516929 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:02.517361 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:02.517365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:02.517369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:02.518649 (+ 1280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:02.518835 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:02.520306 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0923 20:16:02.520471 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16498,"thread_start_us":16674,"threads_started":1}
I20260923 20:16:02.521776 12568 heartbeater.cc:347] Connected to a master server at 127.5.83.125:45061
I20260923 20:16:02.521906 12568 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:02.522554 12568 heartbeater.cc:511] Master 127.5.83.125:45061 requested a full tablet report, sending...
I20260923 20:16:02.524266 12199 ts_manager.cc:194] Registered new tserver with Master: 8eb6ee177604473fa83b0371abc96e51 (127.5.83.65:40173)
I20260923 20:16:02.596232 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 11795
I20260923 20:16:02.612883 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:45889
--local_ip_for_outbound_sockets=127.5.83.66
--tserver_master_addrs=127.5.83.126:42939
--webserver_port=44559
--webserver_interface=127.5.83.66
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab
--principal=oryx/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.83.126:42939,127.5.83.125:45061
--tserver_master_addrs=127.5.83.126:42939,127.5.83.125:45061,127.5.83.124:46391 with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:16:02.853253 12579 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: NON_VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: true } } }
I20260923 20:16:02.853775 12579 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:02.854627 12579 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05: attempting to promote NON_VOTER 2316a8bbd88e4697bcc22dc8943a72d2 to VOTER
I20260923 20:16:02.855173 12579 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } }
I20260923 20:16:02.857218 12209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Refusing update from remote peer bbcb22e21bc74c539336c817d2b64b05: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 20:16:02.857234 12398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 LEARNER]: Refusing update from remote peer bbcb22e21bc74c539336c817d2b64b05: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 20:16:02.857817 12581 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 20:16:02.858237 12580 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 20:16:02.862159 12580 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2316a8bbd88e4697bcc22dc8943a72d2 (127.5.83.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } } }
I20260923 20:16:02.863202 12209 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2316a8bbd88e4697bcc22dc8943a72d2 (127.5.83.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } } }
I20260923 20:16:02.865631 12580 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } } }
I20260923 20:16:02.865883 12580 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:02.866171 12579 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } } }
I20260923 20:16:02.866245 12580 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } } }
I20260923 20:16:02.866415 12579 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:02.866451 12580 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:02.867046 12398 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2316a8bbd88e4697bcc22dc8943a72d2 (127.5.83.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } } }
I20260923 20:16:02.868965 12586 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } } }
I20260923 20:16:02.869364 12586 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:16:02.870723 12587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbcb22e21bc74c539336c817d2b64b05" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } } }
I20260923 20:16:02.871070 12587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 20:16:02.944695 12578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:02.945017 12578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:02.945180 12578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:02.952350 12578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:02.952507 12578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:02.952709 12578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:16:02.962666 12578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab
--principal=oryx/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:45889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=44559
--tserver_master_addrs=127.5.83.126:42939,127.5.83.125:45061,127.5.83.124:46391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:02.964138 12578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:02.965466 12578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:02.978226 12592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:02.980742 12595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:02.980859 12593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:02.981191 12578 server_base.cc:1034] running on GCE node
I20260923 20:16:02.981814 12578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:02.982795 12578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:02.984055 12578 hybrid_clock.cc:648] HybridClock initialized: now 1790194562983981 us; error 47 us; skew 500 ppm
I20260923 20:16:02.984349 12578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:02 dist-test-slave-vl24 krb5kdc[11555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194562, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:02.987460 12578 init.cc:377] Logged in from keytab as oryx/127.5.83.66@KRBTEST.COM (short username oryx)
I20260923 20:16:02.988927 12578 webserver.cc:460] Webserver started at http://127.5.83.66:44559/ using document root <none> and password file <none>
I20260923 20:16:02.989632 12578 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:02.989773 12578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:02.994968 12578 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 20:16:02.997617 12602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:02.999189 12578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:16:02.999347 12578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "cff6b76a733b4b80ba43b8c45a19a60a"
format_stamp: "Formatted at 2026-09-23 20:15:59 on dist-test-slave-vl24"
server_key: "4fc4b03349410c5b210e09ded6634c31"
server_key_iv: "5c18621bef0fd071fdd3af2dff7ce66c"
server_key_version: "encryptionkey@0"
I20260923 20:16:03.000655 12578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:03.022042 12578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:03.026281 12578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:03.026561 12578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:03.027560 12578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:03.029003 12578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:03.029085 12578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:03.029213 12578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:03.029311 12578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:03.063479 12578 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:45889
I20260923 20:16:03.063524 12714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:45889 every 8 connection(s)
I20260923 20:16:03.065050 12578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:16:03.074752 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 12578
I20260923 20:16:03.075060 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 11928
Sep 23 20:16:03 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194562, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.124@KRBTEST.COM
Sep 23 20:16:03 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194562, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.125@KRBTEST.COM
Sep 23 20:16:03 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194562, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
Sep 23 20:16:03 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.67@KRBTEST.COM
I20260923 20:16:03.122030 12720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.068500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:51383 (local address 127.5.83.125:45061)
0923 20:16:03.070567 (+ 2067us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.070577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.101701 (+ 31124us) server_negotiation.cc:407] Connection header received
0923 20:16:03.101878 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.101884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.101963 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.102088 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:03.104197 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.105226 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.108124 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.109179 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.109361 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.114425 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:03.114432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:03.114437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:03.114497 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:03.118047 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.119054 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.119060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.119064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.119205 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.119760 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.119766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.119771 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.120127 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:03.120247 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.121355 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.121619 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1899,"thread_start_us":1746,"threads_started":1}
I20260923 20:16:03.122349 12719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.068146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:52793 (local address 127.5.83.126:42939)
0923 20:16:03.081813 (+ 13667us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.081822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.099157 (+ 17335us) server_negotiation.cc:407] Connection header received
0923 20:16:03.099257 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.099266 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.099399 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.099542 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:03.105838 (+ 6296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.106925 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.109963 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.110639 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.110744 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.115145 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:03.115150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:03.115153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:03.115194 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:03.119168 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.119980 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.119985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.119989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.120141 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.120645 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.120650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.120653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.120963 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:03.121094 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.121580 (+ 486us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.121800 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13490,"thread_start_us":13662,"threads_started":1}
I20260923 20:16:03.123291 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.068911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:46687 (local address 127.5.83.124:46391)
0923 20:16:03.100917 (+ 32006us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.100922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.100938 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:16:03.100984 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.100989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.101046 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.101131 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:03.104074 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.105177 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.108118 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.108823 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.108937 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.114119 (+ 5182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:03.114125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:03.114131 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:03.114184 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:03.118584 (+ 4400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.119407 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.119412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.119416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.119556 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.120044 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.120048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.120054 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.120403 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:03.120533 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.122676 (+ 2143us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.122899 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31876,"thread_start_us":31798,"threads_started":1}
I20260923 20:16:03.123780 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.069262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:45061 (local address 127.5.83.66:51383)
0923 20:16:03.101575 (+ 32313us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:03.101590 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:03.101743 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:03.102158 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:03.102163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:03.102724 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:03.104073 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.104082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.105412 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.105422 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.107940 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.107957 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.109906 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.109914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.110193 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.110775 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:03.110896 (+ 121us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:03.114222 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:03.118204 (+ 3982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:03.118212 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:03.118231 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:03.118903 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:03.119355 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:03.119359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:03.119364 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:03.119627 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:03.120373 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:03.120384 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:03.120838 (+ 454us) client_negotiation.cc:769] Sending connection context
0923 20:16:03.122248 (+ 1410us) client_negotiation.cc:240] Negotiation successful
0923 20:16:03.122532 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32204,"mutex_wait_us":552,"spinlock_wait_cycles":18176,"thread_start_us":30623,"threads_started":1}
I20260923 20:16:03.123934 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.068652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.66:52793)
0923 20:16:03.092026 (+ 23374us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:03.092106 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:03.095787 (+ 3681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:03.103354 (+ 7567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:03.103359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:03.103889 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:03.105625 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.105638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.107103 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.107108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.109778 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.109791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.111086 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.111094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.111292 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.111872 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:03.111903 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:03.114947 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:03.119313 (+ 4366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:03.119321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:03.119326 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:03.119845 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:03.120279 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:03.120284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:03.120287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:03.120477 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:03.121171 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:03.121175 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:03.121314 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 20:16:03.122614 (+ 1300us) client_negotiation.cc:240] Negotiation successful
0923 20:16:03.122944 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22961,"mutex_wait_us":2,"thread_start_us":32556,"threads_started":1}
I20260923 20:16:03.124853 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.068655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:46391 (local address 127.5.83.66:46687)
0923 20:16:03.100305 (+ 31650us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:03.100324 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:03.100450 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:03.101551 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:03.101567 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:03.102541 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:03.103863 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.103892 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.105337 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.105342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.107930 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.107943 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.109324 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.109333 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.109652 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.110415 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:03.110454 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:03.113823 (+ 3369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:03.118748 (+ 4925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:03.118753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:03.118757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:03.119278 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:03.119684 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:03.119690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:03.119693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:03.119891 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:03.120621 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:03.120627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:03.120833 (+ 206us) client_negotiation.cc:769] Sending connection context
0923 20:16:03.121188 (+ 355us) client_negotiation.cc:240] Negotiation successful
0923 20:16:03.121723 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31422,"mutex_wait_us":647,"spinlock_wait_cycles":20224,"thread_start_us":22323,"threads_started":1}
I20260923 20:16:03.124886 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.076835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47624 (local address 127.5.83.67:40049)
0923 20:16:03.089226 (+ 12391us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.089240 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.090190 (+ 950us) server_negotiation.cc:407] Connection header received
0923 20:16:03.090294 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.090307 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.090708 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.090903 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:03.101923 (+ 11020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.103138 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.105066 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.106275 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.106411 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.116272 (+ 9861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:03.116310 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:03.116330 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:03.116375 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:03.120479 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.121488 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.121497 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.121500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.121653 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.122168 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.122173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.122175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.122639 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:03.122993 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.124188 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.124465 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12153,"thread_start_us":11955,"threads_started":1}
I20260923 20:16:03.125600 12715 heartbeater.cc:347] Connected to a master server at 127.5.83.126:42939
I20260923 20:16:03.125707 12717 heartbeater.cc:347] Connected to a master server at 127.5.83.125:45061
I20260923 20:16:03.125906 12717 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:03.125913 12715 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:03.126577 12716 heartbeater.cc:347] Connected to a master server at 127.5.83.124:46391
I20260923 20:16:03.126645 12051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:03.126718 12716 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:03.126979 12717 heartbeater.cc:511] Master 127.5.83.125:45061 requested a full tablet report, sending...
I20260923 20:16:03.127015 12715 heartbeater.cc:511] Master 127.5.83.126:42939 requested a full tablet report, sending...
I20260923 20:16:03.127499 12716 heartbeater.cc:511] Master 127.5.83.124:46391 requested a full tablet report, sending...
I20260923 20:16:03.128767 11600 ts_manager.cc:194] Re-registered known tserver with Master: cff6b76a733b4b80ba43b8c45a19a60a (127.5.83.66:45889)
I20260923 20:16:03.128960 12199 ts_manager.cc:194] Registered new tserver with Master: cff6b76a733b4b80ba43b8c45a19a60a (127.5.83.66:45889)
I20260923 20:16:03.129038 12388 ts_manager.cc:194] Registered new tserver with Master: cff6b76a733b4b80ba43b8c45a19a60a (127.5.83.66:45889)
I20260923 20:16:03.129907 11600 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:52793
I20260923 20:16:03.218395 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 11928
I20260923 20:16:03.234742 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:40049
--local_ip_for_outbound_sockets=127.5.83.67
--tserver_master_addrs=127.5.83.126:42939
--webserver_port=42485
--webserver_interface=127.5.83.67
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab
--principal=oryx/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.83.126:42939,127.5.83.125:45061
--tserver_master_addrs=127.5.83.126:42939,127.5.83.125:45061,127.5.83.124:46391 with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:16:03.523380 12566 heartbeater.cc:503] Master 127.5.83.126:42939 was elected leader, sending a full tablet report...
W20260923 20:16:03.569888 12728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:03.570142 12728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:03.570231 12728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:03.577482 12728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:03.577590 12728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:03.577700 12728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:16:03.588343 12728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab
--principal=oryx/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:40049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=42485
--tserver_master_addrs=127.5.83.126:42939,127.5.83.125:45061,127.5.83.124:46391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:03.589689 12728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:03.591005 12728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:03.604409 12734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:03.604504 12733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:03.604442 12736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:03.605540 12728 server_base.cc:1034] running on GCE node
I20260923 20:16:03.606256 12728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:03.607110 12728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:03.608335 12728 hybrid_clock.cc:648] HybridClock initialized: now 1790194563608285 us; error 43 us; skew 500 ppm
I20260923 20:16:03.608711 12728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:03 dist-test-slave-vl24 krb5kdc[11555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194563, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:03.611923 12728 init.cc:377] Logged in from keytab as oryx/127.5.83.67@KRBTEST.COM (short username oryx)
I20260923 20:16:03.613456 12728 webserver.cc:460] Webserver started at http://127.5.83.67:42485/ using document root <none> and password file <none>
I20260923 20:16:03.614189 12728 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:03.614348 12728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:03.619827 12728 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 20:16:03.622730 12743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:03.624290 12728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:16:03.624423 12728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "8ada6626f6f04d48a8e4624e98acfe6c"
format_stamp: "Formatted at 2026-09-23 20:16:00 on dist-test-slave-vl24"
server_key: "933bcffb2214d2a4156fce4609456058"
server_key_iv: "f3e3acc2ec9da6c2282490fe90a69459"
server_key_version: "encryptionkey@0"
I20260923 20:16:03.625454 12728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:03.651218 12728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:03.655823 12728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:03.656066 12728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:03.657146 12728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:03.658524 12728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:03.658610 12728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:03.658694 12728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:03.658752 12728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:03.692067 12728 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:40049
I20260923 20:16:03.692165 12855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:40049 every 8 connection(s)
I20260923 20:16:03.693565 12728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:16:03.696326 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 12728
Sep 23 20:16:03 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194563, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.124@KRBTEST.COM
Sep 23 20:16:03 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194563, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
Sep 23 20:16:03 dist-test-slave-vl24 krb5kdc[11555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194563, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.125@KRBTEST.COM
I20260923 20:16:03.757515 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.696573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:57643 (local address 127.5.83.124:46391)
0923 20:16:03.717438 (+ 20865us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.717447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.718560 (+ 1113us) server_negotiation.cc:407] Connection header received
0923 20:16:03.720569 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.720577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.720680 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.720801 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:03.724555 (+ 3754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.725653 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.728577 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.729260 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.729391 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.749175 (+ 19784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:03.749198 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:03.749204 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:03.749253 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:03.753589 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.754843 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.754849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.754852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.755108 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.755695 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.755700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.755703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.756062 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:03.756195 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.756925 (+ 730us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.757167 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20694,"thread_start_us":20582,"threads_started":1}
I20260923 20:16:03.758927 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.705025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:46567 (local address 127.5.83.126:42939)
0923 20:16:03.718436 (+ 13411us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.718442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.718464 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:16:03.720274 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.720282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.720384 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.720504 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:03.723078 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.723844 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.727665 (+ 3821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.728144 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.728233 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.741730 (+ 13497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:03.741739 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:03.741747 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:03.741798 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:03.747452 (+ 5654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.749381 (+ 1929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.749389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.749392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.749776 (+ 384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.751378 (+ 1602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.751386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.751392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.751858 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:03.752020 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.758324 (+ 6304us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.758563 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13280,"thread_start_us":13441,"threads_started":1}
I20260923 20:16:03.759727 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.704274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:46391 (local address 127.5.83.67:57643)
0923 20:16:03.717767 (+ 13493us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:03.717838 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:03.720003 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:03.721801 (+ 1798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:03.721807 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:03.722268 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:03.722903 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.722923 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.725811 (+ 2888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.725815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.728402 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.728412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.729430 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.729437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.729624 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.735950 (+ 6326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:03.735981 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:03.740371 (+ 4390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:03.754054 (+ 13683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:03.754061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:03.754064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:03.754679 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:03.755342 (+ 663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:03.755347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:03.755350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:03.755530 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:03.756281 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:03.756285 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:03.756407 (+ 122us) client_negotiation.cc:769] Sending connection context
0923 20:16:03.758268 (+ 1861us) client_negotiation.cc:240] Negotiation successful
0923 20:16:03.758544 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12986,"mutex_wait_us":840,"spinlock_wait_cycles":37504,"thread_start_us":1297,"threads_started":1}
I20260923 20:16:03.762009 12857 heartbeater.cc:347] Connected to a master server at 127.5.83.124:46391
I20260923 20:16:03.762358 12857 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:03.763444 12857 heartbeater.cc:511] Master 127.5.83.124:46391 requested a full tablet report, sending...
I20260923 20:16:03.766247 12388 ts_manager.cc:194] Registered new tserver with Master: 8ada6626f6f04d48a8e4624e98acfe6c (127.5.83.67:40049)
I20260923 20:16:03.781070 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.705540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.67:46567)
0923 20:16:03.718128 (+ 12588us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:03.718152 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:03.719990 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:03.720820 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:03.720832 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:03.721912 (+ 1080us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:03.722909 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.722927 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.723997 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.724005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.727065 (+ 3060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.727085 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.729155 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.729161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.729335 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.735629 (+ 6294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:03.735685 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:03.741435 (+ 5750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:03.747768 (+ 6333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:03.747784 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:03.747813 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:03.749210 (+ 1397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:03.750699 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:03.750705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:03.750711 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:03.751213 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:03.755360 (+ 4147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:03.755368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:03.755916 (+ 548us) client_negotiation.cc:769] Sending connection context
0923 20:16:03.756220 (+ 304us) client_negotiation.cc:240] Negotiation successful
0923 20:16:03.757065 (+ 845us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12450,"mutex_wait_us":97,"spinlock_wait_cycles":5888,"thread_start_us":11192,"threads_started":1}
I20260923 20:16:03.788414 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.696583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:58679 (local address 127.5.83.125:45061)
0923 20:16:03.719118 (+ 22535us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.719125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.720420 (+ 1295us) server_negotiation.cc:407] Connection header received
0923 20:16:03.720612 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.720617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.720702 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.720824 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:03.724443 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.725571 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.727877 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.728764 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.728909 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.778733 (+ 49824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:03.778763 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:03.778778 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:03.778856 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:03.783258 (+ 4402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.784841 (+ 1583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.784851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.784857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.785053 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.785955 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.785963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.785969 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.786341 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:03.786494 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.787577 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.787882 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22402,"thread_start_us":22290,"threads_started":1}
I20260923 20:16:03.789888 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.703743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:45061 (local address 127.5.83.67:58679)
0923 20:16:03.720317 (+ 16574us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:03.720332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:03.720439 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:03.720841 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:03.720846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:03.724024 (+ 3178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:03.724293 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.724304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.725738 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.725742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.727721 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.727734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.729009 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.729016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.729892 (+ 876us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.752623 (+ 22731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:03.752648 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:03.778416 (+ 25768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:03.783477 (+ 5061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:03.783486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:03.783493 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:03.784378 (+ 885us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:03.785296 (+ 918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:03.785303 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:03.785309 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:03.785490 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:03.786631 (+ 1141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:03.786637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:03.786814 (+ 177us) client_negotiation.cc:769] Sending connection context
0923 20:16:03.788878 (+ 2064us) client_negotiation.cc:240] Negotiation successful
0923 20:16:03.789293 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1815,"mutex_wait_us":191,"thread_start_us":13882,"threads_started":1}
I20260923 20:16:03.790980 12858 heartbeater.cc:347] Connected to a master server at 127.5.83.125:45061
I20260923 20:16:03.791188 12858 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:03.791743 12858 heartbeater.cc:511] Master 127.5.83.125:45061 requested a full tablet report, sending...
I20260923 20:16:03.798300 12199 ts_manager.cc:194] Registered new tserver with Master: 8ada6626f6f04d48a8e4624e98acfe6c (127.5.83.67:40049)
I20260923 20:16:03.798373 12856 heartbeater.cc:347] Connected to a master server at 127.5.83.126:42939
I20260923 20:16:03.798522 12856 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:03.799022 12856 heartbeater.cc:511] Master 127.5.83.126:42939 requested a full tablet report, sending...
I20260923 20:16:03.801059 11600 ts_manager.cc:194] Re-registered known tserver with Master: 8ada6626f6f04d48a8e4624e98acfe6c (127.5.83.67:40049)
I20260923 20:16:03.802215 11600 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:46567
I20260923 20:16:03.808651 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.731296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.5.83.124:46391)
0923 20:16:03.758217 (+ 26921us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.758234 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.781678 (+ 23444us) server_negotiation.cc:407] Connection header received
0923 20:16:03.781893 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.781901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.782034 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.782145 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:03.785553 (+ 3408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.786948 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.795128 (+ 8180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.795819 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.795927 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.801051 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:03.801059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:03.801063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:03.801110 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:03.805021 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.806003 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.806011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.806018 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.806185 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.806761 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.806766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.806769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.807172 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:03.807427 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.808045 (+ 618us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.808260 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26798,"thread_start_us":60907,"threads_started":1}
I20260923 20:16:03.819212 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.730727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.5.83.125:45061)
0923 20:16:03.788802 (+ 58075us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.788811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.792998 (+ 4187us) server_negotiation.cc:407] Connection header received
0923 20:16:03.793300 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.793306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.793397 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.793459 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:03.796074 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.797259 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.802647 (+ 5388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.803425 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.803537 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.811839 (+ 8302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:03.811848 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:03.811854 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:03.811898 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:03.815913 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.816998 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.817003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.817006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.817150 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.817606 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.817611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.817614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.817944 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:03.818209 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.818560 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.818793 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57947,"thread_start_us":61266,"threads_started":1}
I20260923 20:16:03.821136 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.730581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.5.83.126:42939)
0923 20:16:03.759166 (+ 28585us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.759171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.798191 (+ 39020us) server_negotiation.cc:407] Connection header received
0923 20:16:03.800293 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.800301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.800398 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.800467 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:03.804817 (+ 4350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.805957 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.810067 (+ 4110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.810698 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.810821 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.813974 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:03.813988 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:03.813993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:03.814033 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:03.817604 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.818515 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.818522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.818525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.818655 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:03.819070 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:03.819074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:03.819077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:03.819420 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:03.819653 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.820062 (+ 409us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.820793 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28478,"thread_start_us":66908,"threads_started":1}
I20260923 20:16:03.825948 11600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:16:03.834751 11600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:16:03.864629 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.853211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:40173 (local address 127.0.0.1:38540)
0923 20:16:03.855158 (+ 1947us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:03.855176 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:03.855282 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:03.857704 (+ 2422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:03.857708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:03.857733 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:03.858220 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.858234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.859871 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.859878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.861623 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.861635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.863538 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.863546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.863856 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.863952 (+ 96us) client_negotiation.cc:769] Sending connection context
0923 20:16:03.864044 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 20:16:03.864187 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1322,"mutex_wait_us":35,"thread_start_us":1472,"threads_started":1}
I20260923 20:16:03.864882 12886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.852821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:45889 (local address 127.0.0.1:35636)
0923 20:16:03.853919 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:03.853943 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:03.854138 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:03.855975 (+ 1837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:03.855982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:03.856019 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:03.856583 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.856600 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.859508 (+ 2908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.859512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.862448 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.862459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.864065 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.864071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.864185 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.864206 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 20:16:03.864261 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 20:16:03.864391 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":42,"thread_start_us":1558,"threads_started":1}
I20260923 20:16:03.865558 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.854039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.5.83.65:40173)
0923 20:16:03.857181 (+ 3142us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.857188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.857209 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:16:03.857281 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.857287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.857511 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.857717 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:03.858469 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.859764 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.861769 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.863271 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.863397 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.863613 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.865080 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.865258 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2993,"thread_start_us":2867,"threads_started":1}
I20260923 20:16:03.866961 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.853634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:40049 (local address 127.0.0.1:47638)
0923 20:16:03.858420 (+ 4786us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:03.858445 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:03.858621 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:03.859201 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:03.859206 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:03.859230 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:03.859642 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.859653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.861800 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.861805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.863909 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.863923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.866136 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.866143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.866342 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.866365 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:16:03.866502 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 20:16:03.866619 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4624,"mutex_wait_us":2,"thread_start_us":4453,"threads_started":1}
I20260923 20:16:03.867046 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.853062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.5.83.66:45889)
0923 20:16:03.855257 (+ 2195us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.855269 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.855304 (+ 35us) server_negotiation.cc:407] Connection header received
0923 20:16:03.855422 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.855433 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.855744 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.855974 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:03.856863 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.859343 (+ 2480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.862646 (+ 3303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.863854 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.866354 (+ 2500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.866565 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.866629 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.866782 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1999,"thread_start_us":1871,"threads_started":1}
I20260923 20:16:03.868718 12889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.854408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.5.83.67:40049)
0923 20:16:03.856743 (+ 2335us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.856755 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.858522 (+ 1767us) server_negotiation.cc:407] Connection header received
0923 20:16:03.858802 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.858810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.859013 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.859191 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:03.859849 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.861694 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.864058 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.865897 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.866045 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.866323 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.866560 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.866719 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2096,"thread_start_us":1946,"threads_started":1}
I20260923 20:16:03.870097 12501 tablet_service.cc:1467] Processing CreateTablet for tablet 414494b6dbec410ca7b7925bdf85462d (DEFAULT_TABLE table=test-table [id=94e6ff58076640ae86f1f009ef059608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:03.870391 12650 tablet_service.cc:1467] Processing CreateTablet for tablet 414494b6dbec410ca7b7925bdf85462d (DEFAULT_TABLE table=test-table [id=94e6ff58076640ae86f1f009ef059608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:03.871574 12650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414494b6dbec410ca7b7925bdf85462d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:03.871600 12501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414494b6dbec410ca7b7925bdf85462d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:03.872587 12791 tablet_service.cc:1467] Processing CreateTablet for tablet 414494b6dbec410ca7b7925bdf85462d (DEFAULT_TABLE table=test-table [id=94e6ff58076640ae86f1f009ef059608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:03.873759 12791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 414494b6dbec410ca7b7925bdf85462d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:03.887778 12891 tablet_bootstrap.cc:492] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a: Bootstrap starting.
I20260923 20:16:03.889384 12892 tablet_bootstrap.cc:492] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51: Bootstrap starting.
I20260923 20:16:03.892984 12890 tablet_bootstrap.cc:492] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c: Bootstrap starting.
I20260923 20:16:03.895274 12891 tablet_bootstrap.cc:654] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:03.895912 12892 tablet_bootstrap.cc:654] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:03.896641 12890 tablet_bootstrap.cc:654] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:03.896741 12891 log.cc:824] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:03.897264 12892 log.cc:824] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:03.897773 12890 log.cc:824] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:03.900285 12890 tablet_bootstrap.cc:492] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c: No bootstrap required, opened a new log
I20260923 20:16:03.900274 12891 tablet_bootstrap.cc:492] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a: No bootstrap required, opened a new log
I20260923 20:16:03.900316 12892 tablet_bootstrap.cc:492] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51: No bootstrap required, opened a new log
I20260923 20:16:03.900609 12892 ts_tablet_manager.cc:1397] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260923 20:16:03.900624 12890 ts_tablet_manager.cc:1397] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260923 20:16:03.900609 12891 ts_tablet_manager.cc:1397] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260923 20:16:03.909821 12892 raft_consensus.cc:348] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } }
I20260923 20:16:03.909708 12890 raft_consensus.cc:348] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } }
I20260923 20:16:03.910164 12890 raft_consensus.cc:374] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:03.910179 12892 raft_consensus.cc:374] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:03.910265 12890 raft_consensus.cc:729] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ada6626f6f04d48a8e4624e98acfe6c, State: Initialized, Role: FOLLOWER
I20260923 20:16:03.910265 12892 raft_consensus.cc:729] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6ee177604473fa83b0371abc96e51, State: Initialized, Role: FOLLOWER
I20260923 20:16:03.910892 12892 consensus_queue.cc:260] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } }
I20260923 20:16:03.910972 12890 consensus_queue.cc:260] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } }
I20260923 20:16:03.912684 12892 ts_tablet_manager.cc:1428] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260923 20:16:03.912421 12891 raft_consensus.cc:348] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } }
I20260923 20:16:03.912964 12891 raft_consensus.cc:374] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:03.913094 12891 raft_consensus.cc:729] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff6b76a733b4b80ba43b8c45a19a60a, State: Initialized, Role: FOLLOWER
I20260923 20:16:03.913880 12891 consensus_queue.cc:260] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } }
I20260923 20:16:03.918608 12715 heartbeater.cc:503] Master 127.5.83.126:42939 was elected leader, sending a full tablet report...
I20260923 20:16:03.919589 12891 ts_tablet_manager.cc:1428] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a: Time spent starting tablet: real 0.019s user 0.015s sys 0.002s
I20260923 20:16:03.921094 12856 heartbeater.cc:503] Master 127.5.83.126:42939 was elected leader, sending a full tablet report...
I20260923 20:16:03.921584 12890 ts_tablet_manager.cc:1428] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c: Time spent starting tablet: real 0.021s user 0.011s sys 0.007s
I20260923 20:16:03.940281 12896 raft_consensus.cc:482] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:16:03.940662 12896 raft_consensus.cc:504] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } }
I20260923 20:16:03.942354 12896 leader_election.cc:288] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cff6b76a733b4b80ba43b8c45a19a60a (127.5.83.66:45889), 8ada6626f6f04d48a8e4624e98acfe6c (127.5.83.67:40049)
I20260923 20:16:03.949502 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.942890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:36585 (local address 127.5.83.66:45889)
0923 20:16:03.943144 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.943152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.943963 (+ 811us) server_negotiation.cc:407] Connection header received
0923 20:16:03.944213 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.944218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.944297 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.944405 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:03.945071 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.945760 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.947688 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.948463 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.948673 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.948770 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.949086 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.949216 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:16:03.949687 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.942632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:45889 (local address 127.5.83.65:36585)
0923 20:16:03.943819 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:03.943839 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:03.944064 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:03.944448 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:03.944451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:03.944476 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:03.944903 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.944920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.945922 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.945926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.947514 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.947525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.948811 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.948815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.948925 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.948946 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 20:16:03.949059 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 20:16:03.949187 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":111,"thread_start_us":706,"threads_started":1}
I20260923 20:16:03.949816 12889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.942783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:46135 (local address 127.5.83.67:40049)
0923 20:16:03.943073 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.943082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.943809 (+ 727us) server_negotiation.cc:407] Connection header received
0923 20:16:03.943963 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.943977 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.944077 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.944166 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:03.944842 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.945740 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.947652 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.948422 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.948521 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.948674 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.949048 (+ 374us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.949582 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 20:16:03.949854 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.942632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:40049 (local address 127.5.83.65:46135)
0923 20:16:03.943685 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:03.943702 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:03.943827 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:03.944198 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:03.944203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:03.944228 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:03.944670 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.944682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.945883 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.945890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.947498 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:03.947509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.948710 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.948716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:03.948877 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.948900 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:16:03.949021 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 20:16:03.949151 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":4,"thread_start_us":632,"threads_started":1}
I20260923 20:16:03.950723 12670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414494b6dbec410ca7b7925bdf85462d" candidate_uuid: "8eb6ee177604473fa83b0371abc96e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" is_pre_election: true
I20260923 20:16:03.950827 12811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414494b6dbec410ca7b7925bdf85462d" candidate_uuid: "8eb6ee177604473fa83b0371abc96e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" is_pre_election: true
I20260923 20:16:03.951287 12670 raft_consensus.cc:2454] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6ee177604473fa83b0371abc96e51 in term 0.
I20260923 20:16:03.951359 12811 raft_consensus.cc:2454] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb6ee177604473fa83b0371abc96e51 in term 0.
I20260923 20:16:03.952041 12457 leader_election.cc:302] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8ada6626f6f04d48a8e4624e98acfe6c, 8eb6ee177604473fa83b0371abc96e51, cff6b76a733b4b80ba43b8c45a19a60a; no voters:
I20260923 20:16:03.952482 12896 raft_consensus.cc:2781] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:16:03.952689 12896 raft_consensus.cc:482] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:16:03.952785 12896 raft_consensus.cc:3037] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:03.955384 12896 raft_consensus.cc:504] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } }
I20260923 20:16:03.956002 12896 leader_election.cc:288] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [CANDIDATE]: Term 1 election: Requested vote from peers cff6b76a733b4b80ba43b8c45a19a60a (127.5.83.66:45889), 8ada6626f6f04d48a8e4624e98acfe6c (127.5.83.67:40049)
I20260923 20:16:03.956435 12670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414494b6dbec410ca7b7925bdf85462d" candidate_uuid: "8eb6ee177604473fa83b0371abc96e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff6b76a733b4b80ba43b8c45a19a60a"
I20260923 20:16:03.956578 12811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "414494b6dbec410ca7b7925bdf85462d" candidate_uuid: "8eb6ee177604473fa83b0371abc96e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ada6626f6f04d48a8e4624e98acfe6c"
I20260923 20:16:03.956792 12670 raft_consensus.cc:3037] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:03.956848 12811 raft_consensus.cc:3037] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:03.960136 12811 raft_consensus.cc:2454] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6ee177604473fa83b0371abc96e51 in term 1.
I20260923 20:16:03.960130 12670 raft_consensus.cc:2454] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb6ee177604473fa83b0371abc96e51 in term 1.
I20260923 20:16:03.960620 12456 leader_election.cc:302] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ada6626f6f04d48a8e4624e98acfe6c, 8eb6ee177604473fa83b0371abc96e51; no voters:
I20260923 20:16:03.960983 12896 raft_consensus.cc:2781] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:03.961544 12896 raft_consensus.cc:686] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 1 LEADER]: Becoming Leader. State: Replica: 8eb6ee177604473fa83b0371abc96e51, State: Running, Role: LEADER
I20260923 20:16:03.962158 12896 consensus_queue.cc:237] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } }
W20260923 20:16:03.964619 12859 tablet.cc:2369] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:16:03.968365 11600 catalog_manager.cc:5495] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eb6ee177604473fa83b0371abc96e51 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "8eb6ee177604473fa83b0371abc96e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } health_report { overall_health: UNKNOWN } } }
W20260923 20:16:03.983003 12569 tablet.cc:2369] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:16:03.992851 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:03.987123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.5.83.65:40173)
0923 20:16:03.987343 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:03.987353 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:03.987457 (+ 104us) server_negotiation.cc:407] Connection header received
0923 20:16:03.987706 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:03.987711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:03.987789 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:03.987949 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:16:03.988416 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.989248 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.990625 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:03.991056 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:03.991200 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:03.991708 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:16:03.992222 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:16:03.992406 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:03.992468 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 20:16:03.992583 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:16:04.002288 12811 raft_consensus.cc:1261] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c [term 1 FOLLOWER]: Refusing update from remote peer 8eb6ee177604473fa83b0371abc96e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:04.002312 12670 raft_consensus.cc:1261] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a [term 1 FOLLOWER]: Refusing update from remote peer 8eb6ee177604473fa83b0371abc96e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:04.003475 12901 consensus_queue.cc:1035] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:04.003877 12896 consensus_queue.cc:1035] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:04.020884 12904 mvcc.cc:205] Tried to move back new op lower bound from 7332636934142001152 to 7332636933993181184. Current Snapshot: MvccSnapshot[applied={T|T < 7332636934142001152}]
I20260923 20:16:04.030012 12906 mvcc.cc:205] Tried to move back new op lower bound from 7332636934142001152 to 7332636933993181184. Current Snapshot: MvccSnapshot[applied={T|T < 7332636934142001152}]
I20260923 20:16:04.042023 11599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:04.042393 11599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:04.049882 11599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbcb22e21bc74c539336c817d2b64b05: Sending DeleteTablet for 3 replicas of tablet 414494b6dbec410ca7b7925bdf85462d
I20260923 20:16:04.051054 12650 tablet_service.cc:1514] Processing DeleteTablet for tablet 414494b6dbec410ca7b7925bdf85462d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:04 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:35636
I20260923 20:16:04.051661 12501 tablet_service.cc:1514] Processing DeleteTablet for tablet 414494b6dbec410ca7b7925bdf85462d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:04 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:38540
I20260923 20:16:04.052204 12913 tablet_replica.cc:330] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a: stopping tablet replica
I20260923 20:16:04.052397 12791 tablet_service.cc:1514] Processing DeleteTablet for tablet 414494b6dbec410ca7b7925bdf85462d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:04 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:47638
I20260923 20:16:04.052865 12913 raft_consensus.cc:2229] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:04.053429 12913 raft_consensus.cc:2258] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:04.056237 12913 ts_tablet_manager.cc:1905] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:04.061282 12913 ts_tablet_manager.cc:1918] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:04.061473 12913 log.cc:1196] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/414494b6dbec410ca7b7925bdf85462d
I20260923 20:16:04.062067 12913 ts_tablet_manager.cc:1939] T 414494b6dbec410ca7b7925bdf85462d P cff6b76a733b4b80ba43b8c45a19a60a: Deleting consensus metadata
I20260923 20:16:04.062292 12914 tablet_replica.cc:330] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51: stopping tablet replica
I20260923 20:16:04.064396 11587 catalog_manager.cc:4841] TS cff6b76a733b4b80ba43b8c45a19a60a (127.5.83.66:45889): tablet 414494b6dbec410ca7b7925bdf85462d (table test-table [id=94e6ff58076640ae86f1f009ef059608]) successfully deleted
I20260923 20:16:04.069011 12914 raft_consensus.cc:2229] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:16:04.069926 12914 raft_consensus.cc:2258] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:04.072728 12914 ts_tablet_manager.cc:1905] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:04.073104 12915 tablet_replica.cc:330] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c: stopping tablet replica
I20260923 20:16:04.079955 12915 raft_consensus.cc:2229] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:04.080801 12915 raft_consensus.cc:2258] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:04.082265 12914 ts_tablet_manager.cc:1918] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:04.082345 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 11568
I20260923 20:16:04.082446 12914 log.cc:1196] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/414494b6dbec410ca7b7925bdf85462d
I20260923 20:16:04.083357 12915 ts_tablet_manager.cc:1905] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:04.083539 11630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:04.083663 12914 ts_tablet_manager.cc:1939] T 414494b6dbec410ca7b7925bdf85462d P 8eb6ee177604473fa83b0371abc96e51: Deleting consensus metadata
I20260923 20:16:04.085525 11584 catalog_manager.cc:4841] TS 8eb6ee177604473fa83b0371abc96e51 (127.5.83.65:40173): tablet 414494b6dbec410ca7b7925bdf85462d (table test-table [id=94e6ff58076640ae86f1f009ef059608]) successfully deleted
I20260923 20:16:04.089152 12915 ts_tablet_manager.cc:1918] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:04.089339 12915 log.cc:1196] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/414494b6dbec410ca7b7925bdf85462d
I20260923 20:16:04.089838 12915 ts_tablet_manager.cc:1939] T 414494b6dbec410ca7b7925bdf85462d P 8ada6626f6f04d48a8e4624e98acfe6c: Deleting consensus metadata
I20260923 20:16:04.100384 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:04.054528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.5.83.125:45061)
0923 20:16:04.054698 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:04.054705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:04.070960 (+ 16255us) server_negotiation.cc:407] Connection header received
0923 20:16:04.071180 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:04.071185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:04.071279 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:04.071389 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:04.073876 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:04.075208 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:04.077400 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:04.078055 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:04.078162 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:04.092891 (+ 14729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:04.092899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:04.092905 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:04.092953 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:04.096715 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:04.097922 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:04.097931 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:04.097936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:04.098143 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:04.098665 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:04.098671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:04.098674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:04.099092 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:04.099366 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:04.099763 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 20:16:04.100012 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 20:16:04.117885 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:04.081164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.5.83.124:46391)
0923 20:16:04.081386 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:04.081393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:04.100486 (+ 19093us) server_negotiation.cc:407] Connection header received
0923 20:16:04.100793 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:04.100800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:04.100888 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:04.100961 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:04.103002 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:04.104202 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:04.106416 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:04.107102 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:04.107207 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:04.110860 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:04.110866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:04.110870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:04.110939 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:04.114615 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:04.115509 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:04.115515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:04.115519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:04.115682 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:04.116174 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:04.116181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:04.116186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:04.116616 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:04.116886 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:04.117284 (+ 398us) server_negotiation.cc:299] Negotiation successful
0923 20:16:04.117529 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:16:04.223469 11586 catalog_manager.cc:4841] TS 8ada6626f6f04d48a8e4624e98acfe6c (127.5.83.67:40049): tablet 414494b6dbec410ca7b7925bdf85462d (table test-table [id=94e6ff58076640ae86f1f009ef059608]) successfully deleted
W20260923 20:16:04.223742 11586 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.5.83.67:40049, user_credentials={real_user=slave}} blocked reactor thread for 132262us
I20260923 20:16:04.224704 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 11568
I20260923 20:16:04.276304 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:04.252181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.5.83.125:45061)
0923 20:16:04.252371 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:04.252377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:04.258975 (+ 6598us) server_negotiation.cc:407] Connection header received
0923 20:16:04.259245 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:04.259251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:04.259339 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:04.259458 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:04.262749 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:04.264044 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:04.266147 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:04.266589 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:04.266745 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:04.270165 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:04.270170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:04.270173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:04.270208 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:04.273556 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:04.274403 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:04.274409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:04.274412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:04.274547 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:04.274924 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:04.274929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:04.274933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:04.275227 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:04.275490 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:04.275824 (+ 334us) server_negotiation.cc:299] Negotiation successful
0923 20:16:04.276030 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:16:04.278733 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:04.251619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.5.83.124:46391)
0923 20:16:04.251854 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:04.251861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:04.262578 (+ 10717us) server_negotiation.cc:407] Connection header received
0923 20:16:04.263098 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:04.263104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:04.263191 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:04.263270 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:04.266092 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:04.267212 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:04.268711 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:04.269156 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:04.269232 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:04.272637 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:04.272642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:04.272645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:04.272676 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:04.275891 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:04.276740 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:04.276746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:04.276750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:04.276896 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:04.277361 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:04.277366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:04.277369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:04.277735 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:04.277956 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:04.278301 (+ 345us) server_negotiation.cc:299] Negotiation successful
0923 20:16:04.278460 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:16:05.029850 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.029064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.67:47343)
0923 20:16:05.029530 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.029650 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":268,"threads_started":1}
W20260923 20:16:05.030323 12856 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:42939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
I20260923 20:16:05.034956 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.034114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.65:44789)
0923 20:16:05.034587 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.034694 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":257,"threads_started":1}
W20260923 20:16:05.035336 12566 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:42939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
I20260923 20:16:05.039482 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.038848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.66:40569)
0923 20:16:05.039232 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.039329 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":210,"threads_started":1}
W20260923 20:16:05.039872 12715 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:42939 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
I20260923 20:16:05.550793 12928 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbcb22e21bc74c539336c817d2b64b05)
I20260923 20:16:05.551043 12928 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } }
I20260923 20:16:05.551139 12929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bbcb22e21bc74c539336c817d2b64b05)
I20260923 20:16:05.551406 12929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } }
I20260923 20:16:05.551962 12928 leader_election.cc:288] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939), 2316a8bbd88e4697bcc22dc8943a72d2 (127.5.83.124:46391)
I20260923 20:16:05.559324 12929 leader_election.cc:288] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939), aa3b4ece5be343a683f8719feb5ab14b (127.5.83.125:45061)
I20260923 20:16:05.561618 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.552119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46316)
0923 20:16:05.560986 (+ 8867us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.561280 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8689,"thread_start_us":8525,"threads_started":1}
W20260923 20:16:05.561965 12186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
I20260923 20:16:05.564692 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.553416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46330)
0923 20:16:05.564022 (+ 10606us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.564356 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10422,"thread_start_us":10227,"threads_started":1}
W20260923 20:16:05.564929 12375 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
I20260923 20:16:05.565853 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.565163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46360)
0923 20:16:05.565627 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.565718 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363}
W20260923 20:16:05.566015 12186 leader_election.cc:334] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939): Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
I20260923 20:16:05.568078 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.567022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46366)
0923 20:16:05.567682 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.567857 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547}
W20260923 20:16:05.568225 12375 leader_election.cc:334] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939): Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
I20260923 20:16:05.571992 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.564645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:45061 (local address 127.0.0.1:56578)
0923 20:16:05.565884 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.565903 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:05.566043 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:05.566666 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:05.566669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:05.566693 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:05.566964 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:05.566979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:05.568790 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:05.568799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:05.570025 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:05.570036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:05.571139 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:05.571144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:05.571332 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:05.571354 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 20:16:05.571475 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 20:16:05.571608 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"spinlock_wait_cycles":768,"thread_start_us":2655,"threads_started":1}
I20260923 20:16:05.572042 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.564607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.5.83.125:45061)
0923 20:16:05.565698 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:05.565703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:05.566163 (+ 460us) server_negotiation.cc:407] Connection header received
0923 20:16:05.566228 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:05.566233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:05.566302 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:05.566400 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:05.567148 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:05.568652 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:05.570184 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:05.570977 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:05.571118 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:05.571229 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:05.571494 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 20:16:05.571641 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":971,"thread_start_us":807,"threads_started":1}
I20260923 20:16:05.572127 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.552387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.5.83.124:46391)
0923 20:16:05.565655 (+ 13268us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:05.565665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:05.565689 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:16:05.565771 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:05.565779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:05.565898 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:05.566696 (+ 798us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:05.567020 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:05.568700 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:05.570292 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:05.571069 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:05.571219 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:05.571332 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:05.571573 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 20:16:05.571708 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13099,"spinlock_wait_cycles":896,"thread_start_us":11805,"threads_started":1}
I20260923 20:16:05.572170 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.552412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:46391 (local address 127.0.0.1:52174)
0923 20:16:05.561380 (+ 8968us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.561414 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:05.561597 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:05.566089 (+ 4492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:05.566096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:05.566127 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:05.566519 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:05.566535 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:05.568872 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:05.568877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:05.570127 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:05.570138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:05.571261 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:05.571267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:05.571425 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:05.571446 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 20:16:05.571550 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:16:05.571735 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8802,"mutex_wait_us":47,"thread_start_us":9348,"threads_started":1}
I20260923 20:16:05.572800 12398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa3b4ece5be343a683f8719feb5ab14b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" is_pre_election: true
I20260923 20:16:05.573046 12209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aa3b4ece5be343a683f8719feb5ab14b" is_pre_election: true
I20260923 20:16:05.573179 12398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa3b4ece5be343a683f8719feb5ab14b in term 1.
I20260923 20:16:05.573359 12209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2316a8bbd88e4697bcc22dc8943a72d2 in term 1.
I20260923 20:16:05.573711 12183 leader_election.cc:302] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2316a8bbd88e4697bcc22dc8943a72d2, aa3b4ece5be343a683f8719feb5ab14b; no voters: bbcb22e21bc74c539336c817d2b64b05
I20260923 20:16:05.573787 12374 leader_election.cc:302] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2316a8bbd88e4697bcc22dc8943a72d2, aa3b4ece5be343a683f8719feb5ab14b; no voters: bbcb22e21bc74c539336c817d2b64b05
I20260923 20:16:05.574002 12928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 20:16:05.574106 12928 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbcb22e21bc74c539336c817d2b64b05)
I20260923 20:16:05.574136 12929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 20:16:05.574187 12928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:16:05.574246 12929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bbcb22e21bc74c539336c817d2b64b05)
I20260923 20:16:05.574338 12929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:16:05.576272 12928 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } }
I20260923 20:16:05.576282 12929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } }
I20260923 20:16:05.576964 12928 leader_election.cc:288] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 2 election: Requested vote from peers bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939), 2316a8bbd88e4697bcc22dc8943a72d2 (127.5.83.124:46391)
I20260923 20:16:05.577005 12929 leader_election.cc:288] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [CANDIDATE]: Term 2 election: Requested vote from peers bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939), aa3b4ece5be343a683f8719feb5ab14b (127.5.83.125:45061)
I20260923 20:16:05.577410 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.577011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46372)
0923 20:16:05.577220 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.577297 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260923 20:16:05.577544 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.577032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46368)
0923 20:16:05.577336 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.577396 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
I20260923 20:16:05.577960 12398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa3b4ece5be343a683f8719feb5ab14b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2316a8bbd88e4697bcc22dc8943a72d2"
I20260923 20:16:05.577966 12209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aa3b4ece5be343a683f8719feb5ab14b"
I20260923 20:16:05.578241 12209 raft_consensus.cc:2379] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2316a8bbd88e4697bcc22dc8943a72d2 in current term 2: Already voted for candidate aa3b4ece5be343a683f8719feb5ab14b in this term.
I20260923 20:16:05.578241 12398 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate aa3b4ece5be343a683f8719feb5ab14b in current term 2: Already voted for candidate 2316a8bbd88e4697bcc22dc8943a72d2 in this term.
I20260923 20:16:05.578588 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.578168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46380)
0923 20:16:05.578358 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.578438 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260923 20:16:05.578809 12186 leader_election.cc:334] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939): Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
I20260923 20:16:05.578902 12186 leader_election.cc:302] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa3b4ece5be343a683f8719feb5ab14b; no voters: 2316a8bbd88e4697bcc22dc8943a72d2, bbcb22e21bc74c539336c817d2b64b05
I20260923 20:16:05.579188 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:05.578410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46388)
0923 20:16:05.578972 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:05.579080 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437}
I20260923 20:16:05.579326 12928 raft_consensus.cc:2726] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260923 20:16:05.579332 12375 leader_election.cc:334] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939): Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
I20260923 20:16:05.579447 12375 leader_election.cc:302] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2316a8bbd88e4697bcc22dc8943a72d2; no voters: aa3b4ece5be343a683f8719feb5ab14b, bbcb22e21bc74c539336c817d2b64b05
I20260923 20:16:05.579703 12929 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 20:16:06.031667 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:06.030925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.67:35623)
0923 20:16:06.031376 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:06.031477 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":238,"threads_started":1}
I20260923 20:16:06.036924 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:06.036028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.65:36129)
0923 20:16:06.036517 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:06.036714 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":274,"threads_started":1}
I20260923 20:16:06.041214 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:06.040512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.66:51777)
0923 20:16:06.040970 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:06.041056 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":206,"threads_started":1}
I20260923 20:16:07.033298 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.032521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.67:46867)
0923 20:16:07.033047 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.033135 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260923 20:16:07.038735 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.037835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.65:59395)
0923 20:16:07.038445 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.038528 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":296,"threads_started":1}
I20260923 20:16:07.042704 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.042034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.66:50691)
0923 20:16:07.042446 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.042533 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":205,"threads_started":1}
I20260923 20:16:07.493130 12944 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:16:07.493438 12944 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } }
I20260923 20:16:07.494328 12944 leader_election.cc:288] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939), 2316a8bbd88e4697bcc22dc8943a72d2 (127.5.83.124:46391)
I20260923 20:16:07.495553 12398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa3b4ece5be343a683f8719feb5ab14b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" is_pre_election: true
I20260923 20:16:07.495707 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.494439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46648)
0923 20:16:07.495324 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.495473 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":449,"threads_started":1}
I20260923 20:16:07.495810 12398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa3b4ece5be343a683f8719feb5ab14b in term 2.
I20260923 20:16:07.496387 12183 leader_election.cc:302] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2316a8bbd88e4697bcc22dc8943a72d2, aa3b4ece5be343a683f8719feb5ab14b; no voters:
I20260923 20:16:07.496759 12944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260923 20:16:07.496903 12944 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:16:07.497052 12944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 2 FOLLOWER]: Advancing to term 3
I20260923 20:16:07.497462 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.496951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46652)
0923 20:16:07.497123 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.497214 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 20:16:07.497687 12186 leader_election.cc:334] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939): Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
I20260923 20:16:07.499332 12944 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } }
I20260923 20:16:07.499884 12944 leader_election.cc:288] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 3 election: Requested vote from peers bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939), 2316a8bbd88e4697bcc22dc8943a72d2 (127.5.83.124:46391)
I20260923 20:16:07.500432 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.500045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46654)
0923 20:16:07.500179 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.500269 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 20:16:07.500437 12398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa3b4ece5be343a683f8719feb5ab14b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2316a8bbd88e4697bcc22dc8943a72d2"
I20260923 20:16:07.500722 12398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 2 FOLLOWER]: Advancing to term 3
I20260923 20:16:07.501361 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.501077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46658)
0923 20:16:07.501200 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.501261 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:16:07.501547 12186 leader_election.cc:334] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939): Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
I20260923 20:16:07.503036 12398 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa3b4ece5be343a683f8719feb5ab14b in term 3.
I20260923 20:16:07.503448 12183 leader_election.cc:302] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2316a8bbd88e4697bcc22dc8943a72d2, aa3b4ece5be343a683f8719feb5ab14b; no voters: bbcb22e21bc74c539336c817d2b64b05
I20260923 20:16:07.503736 12944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 3 FOLLOWER]: Leader election won for term 3
I20260923 20:16:07.504099 12944 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [term 3 LEADER]: Becoming Leader. State: Replica: aa3b4ece5be343a683f8719feb5ab14b, State: Running, Role: LEADER
I20260923 20:16:07.504492 12944 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } }
I20260923 20:16:07.506361 12947 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa3b4ece5be343a683f8719feb5ab14b. Latest consensus state: current_term: 3 leader_uuid: "aa3b4ece5be343a683f8719feb5ab14b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } } }
I20260923 20:16:07.506788 12947 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:07.507409 12949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:16:07.511065 12949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:16:07.511905 12949 catalog_manager.cc:1259] Loaded cluster ID: 8028ca1dcc2a4dd2a2f380dd6787ff86
I20260923 20:16:07.512006 12949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:16:07.513181 12949 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:16:07.513965 12949 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: Loaded TSK: 0
I20260923 20:16:07.514786 12949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:16:07.561810 12198 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:16:07.570186 12198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:16:07.574823 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.574422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46668)
0923 20:16:07.574593 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.574676 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 20:16:07.574896 12398 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [term 3 FOLLOWER]: Refusing update from remote peer aa3b4ece5be343a683f8719feb5ab14b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260923 20:16:07.575673 12947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 20:16:07.575872 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.575488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46684)
0923 20:16:07.575686 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.575748 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 20:16:07.576066 12186 consensus_peers.cc:487] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b -> Peer bbcb22e21bc74c539336c817d2b64b05 (127.5.83.126:42939): Couldn't send request to peer bbcb22e21bc74c539336c817d2b64b05. Status: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 20:16:07.578097 12951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa3b4ece5be343a683f8719feb5ab14b. Latest consensus state: current_term: 3 leader_uuid: "aa3b4ece5be343a683f8719feb5ab14b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } } }
I20260923 20:16:07.578402 12951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:16:07.579857 12944 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "aa3b4ece5be343a683f8719feb5ab14b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } } }
I20260923 20:16:07.580220 12944 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:07.581588 12951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "aa3b4ece5be343a683f8719feb5ab14b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bbcb22e21bc74c539336c817d2b64b05" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 42939 } } peers { permanent_uuid: "aa3b4ece5be343a683f8719feb5ab14b" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 45061 } attrs { promote: false } } peers { permanent_uuid: "2316a8bbd88e4697bcc22dc8943a72d2" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 46391 } attrs { promote: false } } }
I20260923 20:16:07.581811 12951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2316a8bbd88e4697bcc22dc8943a72d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:16:07.594050 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.588048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.5.83.67:40049)
0923 20:16:07.588463 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:07.588467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:07.588482 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:16:07.588593 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:07.588598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:07.588674 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:07.589338 (+ 664us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:07.589523 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.590455 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.592258 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.593104 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.593250 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.593356 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:07.593629 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 20:16:07.593758 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":223,"threads_started":1}
I20260923 20:16:07.594050 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.587737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:40049 (local address 127.0.0.1:47650)
0923 20:16:07.587876 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.587921 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:07.588034 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:07.588844 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:07.588849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:07.588874 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:07.589184 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:07.589193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.590597 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.590602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:07.592108 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:07.592120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.593292 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.593298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:07.593476 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.593499 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:16:07.593605 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:16:07.593735 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 20:16:07.595263 12791 tablet_service.cc:1467] Processing CreateTablet for tablet 0fbe883add744ea3bf6adac323eba80d (DEFAULT_TABLE table=test-table [id=72493349c35b4850b29e50f43d84dde8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:07.595731 12791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fbe883add744ea3bf6adac323eba80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:07.600620 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.589026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:40173 (local address 127.0.0.1:38554)
0923 20:16:07.594275 (+ 5249us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.594292 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:07.594380 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:07.594768 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:07.594774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:07.594801 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:07.594950 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:07.594960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.596351 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.596356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:07.598271 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:07.598284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.599785 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.599791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:07.599999 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.600044 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 20:16:07.600106 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 20:16:07.600230 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5155,"threads_started":1}
I20260923 20:16:07.601649 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.589126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.5.83.65:40173)
0923 20:16:07.589618 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:07.589624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:07.594473 (+ 4849us) server_negotiation.cc:407] Connection header received
0923 20:16:07.594548 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:07.594553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:07.594624 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:07.595099 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:07.595254 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.596196 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.598481 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.599607 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.601074 (+ 1467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.601181 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:07.601228 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 20:16:07.601368 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":264,"threads_started":1}
I20260923 20:16:07.602413 12962 tablet_bootstrap.cc:492] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c: Bootstrap starting.
I20260923 20:16:07.603525 12501 tablet_service.cc:1467] Processing CreateTablet for tablet 0fbe883add744ea3bf6adac323eba80d (DEFAULT_TABLE table=test-table [id=72493349c35b4850b29e50f43d84dde8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:07.603996 12501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fbe883add744ea3bf6adac323eba80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:07.604357 12962 tablet_bootstrap.cc:654] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:07.608279 12962 tablet_bootstrap.cc:492] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c: No bootstrap required, opened a new log
I20260923 20:16:07.608426 12962 ts_tablet_manager.cc:1397] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 20:16:07.609275 12962 raft_consensus.cc:348] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } }
I20260923 20:16:07.609603 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.590418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:45889 (local address 127.0.0.1:35652)
0923 20:16:07.600790 (+ 10372us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.600808 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:07.600905 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:07.602873 (+ 1968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:07.602878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:07.602905 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:07.603137 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:07.603147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.604715 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.604723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:07.606966 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:07.606983 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.608774 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.608780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:07.609000 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.609035 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:16:07.609141 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:16:07.609278 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10276,"thread_start_us":12525,"threads_started":1}
I20260923 20:16:07.609627 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.590777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.5.83.66:45889)
0923 20:16:07.591240 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:07.591246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:07.602543 (+ 11297us) server_negotiation.cc:407] Connection header received
0923 20:16:07.602633 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:07.602638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:07.602719 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:07.603294 (+ 575us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:07.603477 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.604503 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.607357 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.608598 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.608759 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.608867 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:07.609185 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 20:16:07.609343 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":229,"threads_started":1}
I20260923 20:16:07.609648 12962 raft_consensus.cc:374] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:07.609748 12962 raft_consensus.cc:729] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ada6626f6f04d48a8e4624e98acfe6c, State: Initialized, Role: FOLLOWER
I20260923 20:16:07.610016 12962 consensus_queue.cc:260] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } }
I20260923 20:16:07.610224 12964 tablet_bootstrap.cc:492] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51: Bootstrap starting.
I20260923 20:16:07.610404 12650 tablet_service.cc:1467] Processing CreateTablet for tablet 0fbe883add744ea3bf6adac323eba80d (DEFAULT_TABLE table=test-table [id=72493349c35b4850b29e50f43d84dde8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:07.610841 12650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fbe883add744ea3bf6adac323eba80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:07.612603 12964 tablet_bootstrap.cc:654] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:07.615252 12964 tablet_bootstrap.cc:492] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51: No bootstrap required, opened a new log
I20260923 20:16:07.615490 12964 ts_tablet_manager.cc:1397] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 20:16:07.616523 12964 raft_consensus.cc:348] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } }
I20260923 20:16:07.616868 12964 raft_consensus.cc:374] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:07.616927 12962 ts_tablet_manager.cc:1428] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260923 20:16:07.617035 12967 tablet_bootstrap.cc:492] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a: Bootstrap starting.
I20260923 20:16:07.617022 12964 raft_consensus.cc:729] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6ee177604473fa83b0371abc96e51, State: Initialized, Role: FOLLOWER
I20260923 20:16:07.617272 12964 consensus_queue.cc:260] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } }
I20260923 20:16:07.619465 12967 tablet_bootstrap.cc:654] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:07.622948 12858 heartbeater.cc:503] Master 127.5.83.125:45061 was elected leader, sending a full tablet report...
I20260923 20:16:07.623036 12967 tablet_bootstrap.cc:492] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a: No bootstrap required, opened a new log
I20260923 20:16:07.623220 12967 ts_tablet_manager.cc:1397] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 20:16:07.624271 12967 raft_consensus.cc:348] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } }
I20260923 20:16:07.624691 12967 raft_consensus.cc:374] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:07.624861 12967 raft_consensus.cc:729] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff6b76a733b4b80ba43b8c45a19a60a, State: Initialized, Role: FOLLOWER
I20260923 20:16:07.625272 12967 consensus_queue.cc:260] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } }
I20260923 20:16:07.628077 12964 ts_tablet_manager.cc:1428] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260923 20:16:07.629191 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.617661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.67:58849)
0923 20:16:07.628735 (+ 11074us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.628868 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10926,"threads_started":1}
I20260923 20:16:07.632175 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.631837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.67:44015)
0923 20:16:07.631990 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.632064 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:16:07.632913 12967 ts_tablet_manager.cc:1428] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a: Time spent starting tablet: real 0.010s user 0.000s sys 0.004s
I20260923 20:16:07.633213 12717 heartbeater.cc:503] Master 127.5.83.125:45061 was elected leader, sending a full tablet report...
I20260923 20:16:07.635326 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.628114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.65:44889)
0923 20:16:07.633614 (+ 5500us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.633804 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5329,"thread_start_us":5207,"threads_started":1}
I20260923 20:16:07.636340 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.626653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.66:46357)
0923 20:16:07.635917 (+ 9264us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.636037 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9133,"thread_start_us":9135,"threads_started":1}
I20260923 20:16:07.636456 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.636026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.65:45723)
0923 20:16:07.636225 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.636315 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260923 20:16:07.637336 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.637035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.66:33723)
0923 20:16:07.637171 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.637235 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 20:16:07.695847 12965 raft_consensus.cc:482] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:16:07.696102 12965 raft_consensus.cc:504] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } }
I20260923 20:16:07.697988 12965 leader_election.cc:288] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb6ee177604473fa83b0371abc96e51 (127.5.83.65:40173), cff6b76a733b4b80ba43b8c45a19a60a (127.5.83.66:45889)
I20260923 20:16:07.704737 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.698249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:40173 (local address 127.5.83.67:40357)
0923 20:16:07.698798 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.698818 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:07.698940 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:07.699274 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:07.699278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:07.699323 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:07.699658 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:07.699673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.701049 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.701055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:07.702804 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:07.702819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.703976 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.703982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:07.704145 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.704169 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:16:07.704275 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:16:07.704413 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435}
I20260923 20:16:07.704730 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.698614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:40357 (local address 127.5.83.65:40173)
0923 20:16:07.698965 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:07.698973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:07.698996 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:16:07.699069 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:07.699076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:07.699163 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:07.699257 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:07.699816 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.700914 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.702973 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.703780 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.703931 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.704039 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:07.704305 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 20:16:07.704435 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260923 20:16:07.705557 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.698412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:45889 (local address 127.5.83.67:35691)
0923 20:16:07.699979 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.699993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:07.700131 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:07.700458 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:07.700462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:07.700484 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:07.700776 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:07.700784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.702155 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.702160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:07.703567 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:07.703578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.704734 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.704740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:07.704995 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.705030 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:16:07.705164 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 20:16:07.705288 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"thread_start_us":712,"threads_started":1}
I20260923 20:16:07.705740 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.698821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:35691 (local address 127.5.83.66:45889)
0923 20:16:07.699004 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:07.699009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:07.700089 (+ 1080us) server_negotiation.cc:407] Connection header received
0923 20:16:07.700284 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:07.700288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:07.700348 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:07.700452 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:07.700932 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.702014 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.703736 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.704477 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.704683 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.704782 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:07.705211 (+ 429us) server_negotiation.cc:299] Negotiation successful
0923 20:16:07.705383 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 20:16:07.705782 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbe883add744ea3bf6adac323eba80d" candidate_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6ee177604473fa83b0371abc96e51" is_pre_election: true
I20260923 20:16:07.706195 12521 raft_consensus.cc:2454] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ada6626f6f04d48a8e4624e98acfe6c in term 0.
I20260923 20:16:07.706323 12669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbe883add744ea3bf6adac323eba80d" candidate_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" is_pre_election: true
I20260923 20:16:07.706671 12669 raft_consensus.cc:2454] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ada6626f6f04d48a8e4624e98acfe6c in term 0.
I20260923 20:16:07.706899 12744 leader_election.cc:302] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ada6626f6f04d48a8e4624e98acfe6c, 8eb6ee177604473fa83b0371abc96e51; no voters:
I20260923 20:16:07.707304 12965 raft_consensus.cc:2781] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:16:07.707427 12965 raft_consensus.cc:482] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:16:07.707520 12965 raft_consensus.cc:3037] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:07.709708 12965 raft_consensus.cc:504] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } }
I20260923 20:16:07.710325 12965 leader_election.cc:288] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [CANDIDATE]: Term 1 election: Requested vote from peers 8eb6ee177604473fa83b0371abc96e51 (127.5.83.65:40173), cff6b76a733b4b80ba43b8c45a19a60a (127.5.83.66:45889)
I20260923 20:16:07.710781 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbe883add744ea3bf6adac323eba80d" candidate_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb6ee177604473fa83b0371abc96e51"
I20260923 20:16:07.710798 12669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fbe883add744ea3bf6adac323eba80d" candidate_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff6b76a733b4b80ba43b8c45a19a60a"
I20260923 20:16:07.710978 12521 raft_consensus.cc:3037] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:07.710990 12669 raft_consensus.cc:3037] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:07.713140 12521 raft_consensus.cc:2454] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ada6626f6f04d48a8e4624e98acfe6c in term 1.
I20260923 20:16:07.713145 12669 raft_consensus.cc:2454] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ada6626f6f04d48a8e4624e98acfe6c in term 1.
I20260923 20:16:07.713567 12744 leader_election.cc:302] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8ada6626f6f04d48a8e4624e98acfe6c, 8eb6ee177604473fa83b0371abc96e51, cff6b76a733b4b80ba43b8c45a19a60a; no voters:
I20260923 20:16:07.713892 12965 raft_consensus.cc:2781] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:07.714304 12965 raft_consensus.cc:686] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 1 LEADER]: Becoming Leader. State: Replica: 8ada6626f6f04d48a8e4624e98acfe6c, State: Running, Role: LEADER
I20260923 20:16:07.714665 12965 consensus_queue.cc:237] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } } peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } }
I20260923 20:16:07.715289 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.714856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.67:54503)
0923 20:16:07.715088 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.715175 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260923 20:16:07.718356 12198 catalog_manager.cc:5495] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ada6626f6f04d48a8e4624e98acfe6c (127.5.83.67). New cstate: current_term: 1 leader_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ada6626f6f04d48a8e4624e98acfe6c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 } health_report { overall_health: UNKNOWN } } }
I20260923 20:16:07.734823 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.729594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.5.83.67:40049)
0923 20:16:07.729757 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:07.729761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:07.729779 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:16:07.729989 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:07.729993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:07.730072 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:07.730191 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:16:07.730508 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.731201 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.732503 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.732992 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.733093 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.733664 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:16:07.734189 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:16:07.734302 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:07.734441 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 20:16:07.734503 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 20:16:07.738879 12669 raft_consensus.cc:1261] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a [term 1 FOLLOWER]: Refusing update from remote peer 8ada6626f6f04d48a8e4624e98acfe6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:07.739362 12521 raft_consensus.cc:1261] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51 [term 1 FOLLOWER]: Refusing update from remote peer 8ada6626f6f04d48a8e4624e98acfe6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:07.739665 12975 consensus_queue.cc:1035] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff6b76a733b4b80ba43b8c45a19a60a" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 45889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:07.740222 12965 consensus_queue.cc:1035] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb6ee177604473fa83b0371abc96e51" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:07.742633 12568 heartbeater.cc:503] Master 127.5.83.125:45061 was elected leader, sending a full tablet report...
I20260923 20:16:07.743141 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.742548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.65:60569)
0923 20:16:07.742815 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.742944 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260923 20:16:07.743829 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.741879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.66:57315)
0923 20:16:07.742189 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.743647 (+ 1458us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260923 20:16:07.752748 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.752178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.67:44935)
0923 20:16:07.752408 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.752516 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260923 20:16:07.756086 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.748732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.66:41331)
0923 20:16:07.755805 (+ 7073us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.755906 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6951}
I20260923 20:16:07.760675 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.760168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.67:32823)
0923 20:16:07.760336 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.760428 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:16:07.763621 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.763207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.5.83.65:50581)
0923 20:16:07.763358 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:07.763474 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 20:16:07.765476 12199 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:07.765684 12199 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:07.771451 12199 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa3b4ece5be343a683f8719feb5ab14b: Sending DeleteTablet for 3 replicas of tablet 0fbe883add744ea3bf6adac323eba80d
I20260923 20:16:07.772289 12791 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fbe883add744ea3bf6adac323eba80d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:07 UTC) from {username='oryx', principal='oryx/127.5.83.125@KRBTEST.COM'} at 127.0.0.1:47650
I20260923 20:16:07.772850 12650 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fbe883add744ea3bf6adac323eba80d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:07 UTC) from {username='oryx', principal='oryx/127.5.83.125@KRBTEST.COM'} at 127.0.0.1:35652
I20260923 20:16:07.773568 12501 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fbe883add744ea3bf6adac323eba80d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:07 UTC) from {username='oryx', principal='oryx/127.5.83.125@KRBTEST.COM'} at 127.0.0.1:38554
I20260923 20:16:07.773609 12500 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fbe883add744ea3bf6adac323eba80d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:07 UTC) from {username='oryx', principal='oryx/127.5.83.125@KRBTEST.COM'} at 127.0.0.1:38554
I20260923 20:16:07.775429 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 12370
I20260923 20:16:07.787858 12989 tablet_replica.cc:330] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a: stopping tablet replica
I20260923 20:16:07.788093 12989 raft_consensus.cc:2229] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:07.788414 12989 raft_consensus.cc:2258] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:07.789837 12989 ts_tablet_manager.cc:1905] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:07.792929 12988 tablet_replica.cc:330] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c: stopping tablet replica
I20260923 20:16:07.793283 12988 raft_consensus.cc:2229] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:16:07.794201 12988 raft_consensus.cc:2258] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:07.795612 12988 ts_tablet_manager.cc:1905] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:07.797117 12990 tablet_replica.cc:330] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51: stopping tablet replica
I20260923 20:16:07.797344 12990 raft_consensus.cc:2229] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:07.797473 12989 ts_tablet_manager.cc:1918] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:07.797617 12989 log.cc:1196] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/0fbe883add744ea3bf6adac323eba80d
I20260923 20:16:07.797683 12990 raft_consensus.cc:2258] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:07.798099 12989 ts_tablet_manager.cc:1939] T 0fbe883add744ea3bf6adac323eba80d P cff6b76a733b4b80ba43b8c45a19a60a: Deleting consensus metadata
I20260923 20:16:07.799388 12186 catalog_manager.cc:4841] TS cff6b76a733b4b80ba43b8c45a19a60a (127.5.83.66:45889): tablet 0fbe883add744ea3bf6adac323eba80d (table test-table [id=72493349c35b4850b29e50f43d84dde8]) successfully deleted
I20260923 20:16:07.799805 12990 ts_tablet_manager.cc:1905] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:07.802496 12988 ts_tablet_manager.cc:1918] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:07.802642 12988 log.cc:1196] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/0fbe883add744ea3bf6adac323eba80d
I20260923 20:16:07.803073 12988 ts_tablet_manager.cc:1939] T 0fbe883add744ea3bf6adac323eba80d P 8ada6626f6f04d48a8e4624e98acfe6c: Deleting consensus metadata
I20260923 20:16:07.804147 12185 catalog_manager.cc:4841] TS 8ada6626f6f04d48a8e4624e98acfe6c (127.5.83.67:40049): tablet 0fbe883add744ea3bf6adac323eba80d (table test-table [id=72493349c35b4850b29e50f43d84dde8]) successfully deleted
I20260923 20:16:07.805596 12990 ts_tablet_manager.cc:1918] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:07.805754 12990 log.cc:1196] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/0fbe883add744ea3bf6adac323eba80d
I20260923 20:16:07.806149 12990 ts_tablet_manager.cc:1939] T 0fbe883add744ea3bf6adac323eba80d P 8eb6ee177604473fa83b0371abc96e51: Deleting consensus metadata
I20260923 20:16:07.807173 12183 catalog_manager.cc:4841] TS 8eb6ee177604473fa83b0371abc96e51 (127.5.83.65:40173): tablet 0fbe883add744ea3bf6adac323eba80d (table test-table [id=72493349c35b4850b29e50f43d84dde8]) successfully deleted
W20260923 20:16:07.807605 12183 catalog_manager.cc:4808] TS 8eb6ee177604473fa83b0371abc96e51 (127.5.83.65:40173): delete failed for tablet 0fbe883add744ea3bf6adac323eba80d because the tablet was not found. No further retry: Not found: Tablet not found: 0fbe883add744ea3bf6adac323eba80d
I20260923 20:16:07.813651 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.776762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38570 (local address 127.5.83.65:40173)
0923 20:16:07.776972 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:07.776981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:07.798436 (+ 21455us) server_negotiation.cc:407] Connection header received
0923 20:16:07.798614 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:07.798619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:07.798713 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:07.798837 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:07.800859 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.802033 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.803853 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.804558 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.804683 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.808061 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:07.808072 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:07.808085 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:07.808114 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:07.810930 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:07.811708 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:07.811715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:07.811717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:07.811840 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:07.812234 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:07.812237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:07.812239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:07.812624 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:07.812834 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:07.813148 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 20:16:07.813326 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:16:07.814333 12561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:07.935254 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 12370
I20260923 20:16:07.958465 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 12578
I20260923 20:16:07.973469 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:07.959287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.5.83.66:45889)
0923 20:16:07.959488 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:07.959494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:07.959515 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:16:07.959730 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:07.959739 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:07.959805 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:07.959913 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:07.961342 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.962065 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.963304 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:07.964014 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:07.964143 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:07.966803 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:07.966820 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:07.966839 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:07.966891 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:07.970522 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:07.971197 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:07.971207 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:07.971210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:07.971369 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:07.971720 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:07.971725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:07.971729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:07.972169 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:07.972466 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:07.972862 (+ 396us) server_negotiation.cc:299] Negotiation successful
0923 20:16:07.973092 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:16:07.974309 12710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:08.086063 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:08.085488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46698)
0923 20:16:08.085787 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:08.085889 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260923 20:16:08.087766 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:08.087287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:42939 (local address 127.0.0.1:46704)
0923 20:16:08.087506 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:08.087611 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260923 20:16:08.091197 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 12578
I20260923 20:16:08.109560 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 12728
I20260923 20:16:08.124080 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:08.110602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.5.83.67:40049)
0923 20:16:08.110814 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:08.110822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:08.110842 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:16:08.110905 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:08.110909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:08.110970 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:08.111093 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:08.112492 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:08.113347 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:08.114590 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:08.115053 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:08.115201 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:08.118008 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:08.118025 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:08.118042 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:08.118082 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:08.121285 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:08.121974 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:08.121981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:08.121983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:08.122127 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:08.122504 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:08.122509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:08.122513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:08.122904 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:08.123227 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:08.123528 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 20:16:08.123757 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:16:08.125109 12851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:08.256639 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 12728
I20260923 20:16:08.278258 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 12061
I20260923 20:16:08.279381 12229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:08.403542 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 12061
I20260923 20:16:08.425194 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 12162
I20260923 20:16:08.426362 12418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:08.544001 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 12162
2026-09-23T20:16:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11525 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:16:08 dist-test-slave-vl24 krb5kdc[13000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:16:08 dist-test-slave-vl24 krb5kdc[13000](info): set up 2 sockets
Sep 23 20:16:08 dist-test-slave-vl24 krb5kdc[13000](info): commencing operation
krb5kdc: starting...
W20260923 20:16:09.646931 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:16:09 dist-test-slave-vl24 krb5kdc[13000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:16:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.126.keytab.
Entry for principal oryx/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.126.keytab.
I20260923 20:16:09.874073 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:41577
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.126.keytab
--principal=oryx/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:41577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:10.205323 13013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:10.205605 13013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:10.205775 13013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:10.212927 13013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:16:10.213025 13013 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 20:16:10.213106 13013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:10.213225 13013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:10.213287 13013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:16:10.213392 13013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:16:10.223063 13013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:41577
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.126.keytab
--principal=oryx/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:41577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:10.224390 13013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:10.225720 13013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:10.237334 13018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:10.237294 13019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:10.237423 13021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:10.238332 13013 server_base.cc:1034] running on GCE node
I20260923 20:16:10.239106 13013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:10.240830 13013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:10.242110 13013 hybrid_clock.cc:648] HybridClock initialized: now 1790194570242056 us; error 26 us; skew 500 ppm
I20260923 20:16:10.242443 13013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:10 dist-test-slave-vl24 krb5kdc[13000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194570, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:10.252264 13013 init.cc:377] Logged in from keytab as oryx/127.5.83.126@KRBTEST.COM (short username oryx)
I20260923 20:16:10.253969 13013 webserver.cc:460] Webserver started at http://127.5.83.126:45355/ using document root <none> and password file <none>
I20260923 20:16:10.254736 13013 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:10.254875 13013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:10.255227 13013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:10.258342 13013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "2a3618d015a240e3a4bb3d4422ad11bb"
format_stamp: "Formatted at 2026-09-23 20:16:10 on dist-test-slave-vl24"
server_key: "7d2b1524b2c531d9e093e0333b1d8b36"
server_key_iv: "bb2da11ca5efe2295332585aeafa61b0"
server_key_version: "encryptionkey@0"
I20260923 20:16:10.259168 13013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "2a3618d015a240e3a4bb3d4422ad11bb"
format_stamp: "Formatted at 2026-09-23 20:16:10 on dist-test-slave-vl24"
server_key: "7d2b1524b2c531d9e093e0333b1d8b36"
server_key_iv: "bb2da11ca5efe2295332585aeafa61b0"
server_key_version: "encryptionkey@0"
I20260923 20:16:10.264405 13013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:16:10.267774 13028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:10.269313 13013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:16:10.269512 13013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "2a3618d015a240e3a4bb3d4422ad11bb"
format_stamp: "Formatted at 2026-09-23 20:16:10 on dist-test-slave-vl24"
server_key: "7d2b1524b2c531d9e093e0333b1d8b36"
server_key_iv: "bb2da11ca5efe2295332585aeafa61b0"
server_key_version: "encryptionkey@0"
I20260923 20:16:10.269750 13013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:10.291543 13013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:10.295558 13013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:10.295840 13013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:10.318017 13013 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:41577
I20260923 20:16:10.318040 13089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:41577 every 8 connection(s)
I20260923 20:16:10.319429 13013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:16:10.323328 13090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:10.326521 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 13013
I20260923 20:16:10.326745 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:16:10.327574 5453 external_mini_cluster.cc:1428] Setting key 57013f0e98ef1bf3cab9ca191137a11c
I20260923 20:16:10.336503 13090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb: Bootstrap starting.
I20260923 20:16:10.340366 13090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:10.341755 13090 log.cc:824] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:16:10 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:16:10.344854 13090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb: No bootstrap required, opened a new log
I20260923 20:16:10.351076 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:10.329744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.5.83.126:41577)
0923 20:16:10.330577 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:10.330589 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:10.330629 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:16:10.332337 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:10.332361 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:10.332836 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:10.333171 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:10.334919 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:10.336175 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:10.337899 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:10.338585 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:10.338715 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:10.343692 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:10.343700 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:10.343720 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:10.343760 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:10.346942 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:10.347766 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:10.347772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:10.347778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:10.347944 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:10.348321 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:10.348324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:10.348326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:10.348883 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:10.349131 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:10.349570 (+ 439us) server_negotiation.cc:299] Negotiation successful
0923 20:16:10.349983 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":442,"threads_started":1}
I20260923 20:16:10.354126 13090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3618d015a240e3a4bb3d4422ad11bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41577 } }
I20260923 20:16:10.354529 13090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:10.354633 13090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3618d015a240e3a4bb3d4422ad11bb, State: Initialized, Role: FOLLOWER
I20260923 20:16:10.355208 13090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3618d015a240e3a4bb3d4422ad11bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41577 } }
I20260923 20:16:10.355456 13090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:16:10.355597 13090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:16:10.355763 13090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:10.358044 13090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3618d015a240e3a4bb3d4422ad11bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41577 } }
I20260923 20:16:10.358556 13090 leader_election.cc:302] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a3618d015a240e3a4bb3d4422ad11bb; no voters:
I20260923 20:16:10.359153 13090 leader_election.cc:288] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:16:10.359294 13095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:10.360786 13095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [term 1 LEADER]: Becoming Leader. State: Replica: 2a3618d015a240e3a4bb3d4422ad11bb, State: Running, Role: LEADER
I20260923 20:16:10.361331 13095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3618d015a240e3a4bb3d4422ad11bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41577 } }
I20260923 20:16:10.361726 13090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:16:10.364054 13097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a3618d015a240e3a4bb3d4422ad11bb. Latest consensus state: current_term: 1 leader_uuid: "2a3618d015a240e3a4bb3d4422ad11bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3618d015a240e3a4bb3d4422ad11bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41577 } } }
I20260923 20:16:10.364460 13097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:10.365101 13096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a3618d015a240e3a4bb3d4422ad11bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3618d015a240e3a4bb3d4422ad11bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41577 } } }
I20260923 20:16:10.365458 13096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:10.366740 13104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:16:10.372439 13104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:16:10.382496 13104 catalog_manager.cc:1347] Generated new cluster ID: f816606e723d40eaa623482ece22251c
I20260923 20:16:10.382630 13104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:16:10.425369 13104 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:16:10.426716 13104 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:16:10.437305 13104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb: Generated new TSK 0
I20260923 20:16:10.438114 13104 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab.
Entry for principal oryx/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab.
I20260923 20:16:10.500679 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41577
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab
--principal=oryx/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:10.826903 13116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:10.827184 13116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:10.827356 13116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:10.834432 13116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:10.834539 13116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:10.834681 13116 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:16:10.834805 13116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:16:10.844791 13116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab
--principal=oryx/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.126:41577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:10.846114 13116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:10.847463 13116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:10.860914 13124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:10.861042 13121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:10.861053 13122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:10.862339 13116 server_base.cc:1034] running on GCE node
I20260923 20:16:10.863045 13116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:10.864137 13116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:10.865370 13116 hybrid_clock.cc:648] HybridClock initialized: now 1790194570865296 us; error 53 us; skew 500 ppm
I20260923 20:16:10.865744 13116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:10 dist-test-slave-vl24 krb5kdc[13000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194570, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:10.868873 13116 init.cc:377] Logged in from keytab as oryx/127.5.83.65@KRBTEST.COM (short username oryx)
I20260923 20:16:10.870230 13116 webserver.cc:460] Webserver started at http://127.5.83.65:40391/ using document root <none> and password file <none>
I20260923 20:16:10.870882 13116 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:10.870977 13116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:10.871251 13116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:10.874353 13116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "3efea12b7cbb41ada3f316cab53be5b9"
format_stamp: "Formatted at 2026-09-23 20:16:10 on dist-test-slave-vl24"
server_key: "be858ed635b2c6cdfcfe97c79da67ef5"
server_key_iv: "d61937e4139d7ef001eee60e9cb1407c"
server_key_version: "encryptionkey@0"
I20260923 20:16:10.875116 13116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "3efea12b7cbb41ada3f316cab53be5b9"
format_stamp: "Formatted at 2026-09-23 20:16:10 on dist-test-slave-vl24"
server_key: "be858ed635b2c6cdfcfe97c79da67ef5"
server_key_iv: "d61937e4139d7ef001eee60e9cb1407c"
server_key_version: "encryptionkey@0"
I20260923 20:16:10.880472 13116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:16:10.884143 13131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:10.885609 13116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:16:10.885785 13116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "3efea12b7cbb41ada3f316cab53be5b9"
format_stamp: "Formatted at 2026-09-23 20:16:10 on dist-test-slave-vl24"
server_key: "be858ed635b2c6cdfcfe97c79da67ef5"
server_key_iv: "d61937e4139d7ef001eee60e9cb1407c"
server_key_version: "encryptionkey@0"
I20260923 20:16:10.885985 13116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:10.905962 13116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:10.910250 13116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:10.910485 13116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:10.913177 13116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:10.913292 13116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:10.913457 13116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:10.913555 13116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:16:10 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194570, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:16:10.944751 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:10.914297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:57041 (local address 127.5.83.126:41577)
0923 20:16:10.914857 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:10.914866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:10.916364 (+ 1498us) server_negotiation.cc:407] Connection header received
0923 20:16:10.919835 (+ 3471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:10.919842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:10.919978 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:10.920123 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:10.923017 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:10.924196 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:10.927558 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:10.928140 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:10.928295 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:10.935247 (+ 6952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:10.935256 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:10.935281 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:10.935335 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:10.939510 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:10.940971 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:10.940979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:10.940983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:10.941145 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:10.941960 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:10.941967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:10.941973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:10.942384 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:10.942559 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:10.943957 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0923 20:16:10.944241 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
I20260923 20:16:10.946816 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:10.915008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41577 (local address 127.5.83.65:57041)
0923 20:16:10.916118 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:10.916179 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:10.919433 (+ 3254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:10.920504 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:10.920546 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:10.921626 (+ 1080us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:10.922773 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:10.922803 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:10.924409 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:10.924415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:10.927387 (+ 2972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:10.927406 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:10.928502 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:10.928514 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:10.929014 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:10.930122 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:10.930172 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:10.934884 (+ 4712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:10.939735 (+ 4851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:10.939753 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:10.939780 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:10.940784 (+ 1004us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:10.941305 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:10.941314 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:10.941322 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:10.941790 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:10.942564 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:10.942575 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:10.943292 (+ 717us) client_negotiation.cc:769] Sending connection context
0923 20:16:10.943850 (+ 558us) client_negotiation.cc:240] Negotiation successful
0923 20:16:10.944725 (+ 875us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":515,"threads_started":1}
I20260923 20:16:10.956038 13250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:36757 every 8 connection(s)
I20260923 20:16:10.956050 13116 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:36757
I20260923 20:16:10.958072 13116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:16:10.961047 13252 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41577
I20260923 20:16:10.961320 13252 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:10.962244 13252 heartbeater.cc:511] Master 127.5.83.126:41577 requested a full tablet report, sending...
I20260923 20:16:10.965024 13045 ts_manager.cc:194] Registered new tserver with Master: 3efea12b7cbb41ada3f316cab53be5b9 (127.5.83.65:36757)
I20260923 20:16:10.965739 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 13116
I20260923 20:16:10.965972 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:16:10.966639 5453 external_mini_cluster.cc:1428] Setting key 94afa4fc1f98ece7d6d4bdedb78c54df
I20260923 20:16:10.967239 13045 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:57041
I20260923 20:16:10.971446 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:10.956209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:46347 (local address 127.5.83.126:41577)
0923 20:16:10.956457 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:10.956465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:10.956904 (+ 439us) server_negotiation.cc:407] Connection header received
0923 20:16:10.957076 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:10.957082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:10.957187 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:10.957332 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:10.958359 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:10.959182 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:10.960968 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:10.961587 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:10.961705 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:10.964000 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:10.964007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:10.964012 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:10.964052 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:10.967563 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:10.969302 (+ 1739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:10.969309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:10.969313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:10.969469 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:10.969932 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:10.969937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:10.969940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:10.970292 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:10.970443 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:10.971002 (+ 559us) server_negotiation.cc:299] Negotiation successful
0923 20:16:10.971206 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 20:16:10.971787 13251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:10.955605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41577 (local address 127.5.83.65:46347)
0923 20:16:10.956785 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:10.956801 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:10.956941 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:10.957378 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:10.957382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:10.957825 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:10.958191 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:10.958206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:10.959371 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:10.959376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:10.960728 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:10.960740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:10.961795 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:10.961802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:10.961989 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:10.962771 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:10.962792 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:10.963758 (+ 966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:10.968523 (+ 4765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:10.968529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:10.968532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:10.969161 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:10.969604 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:10.969610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:10.969614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:10.969792 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:10.970432 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:10.970441 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:10.970673 (+ 232us) client_negotiation.cc:769] Sending connection context
0923 20:16:10.970832 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 20:16:10.971227 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"thread_start_us":765,"threads_started":1}
WARNING: no policy specified for oryx/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab.
Entry for principal oryx/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab.
I20260923 20:16:11.019469 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41577
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab
--principal=oryx/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:11.350749 13256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:11.351118 13256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:11.351292 13256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:11.358592 13256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:11.358760 13256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:11.358911 13256 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:16:11.359056 13256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:16:11.369222 13256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab
--principal=oryx/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.126:41577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:11.370577 13256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:11.371969 13256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:11.385872 13262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:11.385929 13261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:11.386022 13264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:11.386732 13256 server_base.cc:1034] running on GCE node
I20260923 20:16:11.387404 13256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:11.388383 13256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:11.389622 13256 hybrid_clock.cc:648] HybridClock initialized: now 1790194571389561 us; error 41 us; skew 500 ppm
I20260923 20:16:11.389940 13256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:11 dist-test-slave-vl24 krb5kdc[13000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194571, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:11.393618 13256 init.cc:377] Logged in from keytab as oryx/127.5.83.66@KRBTEST.COM (short username oryx)
I20260923 20:16:11.395213 13256 webserver.cc:460] Webserver started at http://127.5.83.66:44415/ using document root <none> and password file <none>
I20260923 20:16:11.395911 13256 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:11.396030 13256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:11.396377 13256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:11.399407 13256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "50ea93ba0f394262814ef7f6202be338"
format_stamp: "Formatted at 2026-09-23 20:16:11 on dist-test-slave-vl24"
server_key: "97fc818e4ea3f2bbb3ba33a3086ae916"
server_key_iv: "dfdf9c58528bcd4d786e810a199aba99"
server_key_version: "encryptionkey@0"
I20260923 20:16:11.400221 13256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "50ea93ba0f394262814ef7f6202be338"
format_stamp: "Formatted at 2026-09-23 20:16:11 on dist-test-slave-vl24"
server_key: "97fc818e4ea3f2bbb3ba33a3086ae916"
server_key_iv: "dfdf9c58528bcd4d786e810a199aba99"
server_key_version: "encryptionkey@0"
I20260923 20:16:11.405647 13256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:16:11.409147 13271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:11.410485 13256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:16:11.410648 13256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "50ea93ba0f394262814ef7f6202be338"
format_stamp: "Formatted at 2026-09-23 20:16:11 on dist-test-slave-vl24"
server_key: "97fc818e4ea3f2bbb3ba33a3086ae916"
server_key_iv: "dfdf9c58528bcd4d786e810a199aba99"
server_key_version: "encryptionkey@0"
I20260923 20:16:11.410841 13256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:11.437801 13256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:11.442425 13256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:11.442716 13256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:11.445595 13256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:11.445703 13256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:11.445847 13256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:11.445919 13256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:16:11 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194571, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:16:11.471083 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:11.445959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:36299 (local address 127.5.83.126:41577)
0923 20:16:11.446105 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:11.446109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:11.447539 (+ 1430us) server_negotiation.cc:407] Connection header received
0923 20:16:11.449833 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:11.449839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:11.449933 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:11.450044 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:11.452357 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:11.453418 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:11.455999 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:11.456785 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:11.456925 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:11.461672 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:11.461683 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:11.461692 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:11.461748 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:11.466014 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:11.467393 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:11.467401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:11.467408 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:11.467590 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:11.468312 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:11.468318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:11.468323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:11.468815 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:11.468989 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:11.470329 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0923 20:16:11.470620 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 20:16:11.473158 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:11.446603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41577 (local address 127.5.83.66:36299)
0923 20:16:11.447375 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:11.447414 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:11.449563 (+ 2149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:11.450292 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:11.450303 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:11.451067 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:11.452182 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:11.452203 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:11.453574 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:11.453580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:11.455816 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:11.455837 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:11.456994 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:11.456999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:11.457305 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:11.458143 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:11.458164 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:11.461320 (+ 3156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:11.466209 (+ 4889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:11.466225 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:11.466254 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:11.467219 (+ 965us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:11.467753 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:11.467765 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:11.467769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:11.468162 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:11.469006 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:11.469020 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:11.469726 (+ 706us) client_negotiation.cc:769] Sending connection context
0923 20:16:11.470185 (+ 459us) client_negotiation.cc:240] Negotiation successful
0923 20:16:11.470968 (+ 783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":350,"threads_started":1}
I20260923 20:16:11.494724 13256 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:40643
I20260923 20:16:11.494743 13390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:40643 every 8 connection(s)
I20260923 20:16:11.496259 13256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:16:11.498850 13391 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41577
I20260923 20:16:11.499104 13391 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:11.499742 13391 heartbeater.cc:511] Master 127.5.83.126:41577 requested a full tablet report, sending...
I20260923 20:16:11.501250 13044 ts_manager.cc:194] Registered new tserver with Master: 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643)
I20260923 20:16:11.502526 13044 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:36299
I20260923 20:16:11.503141 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 13256
I20260923 20:16:11.503347 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:16:11.503811 5453 external_mini_cluster.cc:1428] Setting key bdd6aba46489d891999019892240c33c
I20260923 20:16:11.505231 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:11.489349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41577 (local address 127.5.83.66:52739)
0923 20:16:11.490865 (+ 1516us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:11.490887 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:11.491072 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:11.491540 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:11.491552 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:11.492001 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:11.492308 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:11.492318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:11.493863 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:11.493871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:11.495661 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:11.495676 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:11.496491 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:11.496498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:11.496727 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:11.497554 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:11.497580 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:11.498683 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:11.502383 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:11.502387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:11.502390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:11.502824 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:11.503275 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:11.503281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:11.503284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:11.503448 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:11.504156 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:11.504161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:11.504416 (+ 255us) client_negotiation.cc:769] Sending connection context
0923 20:16:11.504618 (+ 202us) client_negotiation.cc:240] Negotiation successful
0923 20:16:11.504911 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"mutex_wait_us":39,"thread_start_us":1174,"threads_started":1}
I20260923 20:16:11.506124 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:11.489202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:52739 (local address 127.5.83.126:41577)
0923 20:16:11.489425 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:11.489433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:11.490992 (+ 1559us) server_negotiation.cc:407] Connection header received
0923 20:16:11.491247 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:11.491255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:11.491369 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:11.491497 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:11.492466 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:11.493642 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:11.495822 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:11.496265 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:11.496428 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:11.498870 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:11.498874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:11.498878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:11.498910 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:11.502268 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:11.502946 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:11.502953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:11.502956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:11.503135 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:11.503582 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:11.503588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:11.503591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:11.504014 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:11.504175 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:11.505603 (+ 1428us) server_negotiation.cc:299] Negotiation successful
0923 20:16:11.505835 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for oryx/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab.
Entry for principal oryx/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab.
I20260923 20:16:11.556408 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41577
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab
--principal=oryx/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:11.884742 13395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:11.885087 13395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:11.885260 13395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:11.892644 13395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:11.892793 13395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:11.892959 13395 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 20:16:11.893100 13395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:16:11.902671 13395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab
--principal=oryx/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.83.126:41577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:11.903985 13395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:11.905342 13395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:11.918869 13400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:11.918888 13401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:11.919072 13403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:11.919639 13395 server_base.cc:1034] running on GCE node
I20260923 20:16:11.920250 13395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:11.921250 13395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:11.922487 13395 hybrid_clock.cc:648] HybridClock initialized: now 1790194571922406 us; error 61 us; skew 500 ppm
I20260923 20:16:11.922785 13395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:11 dist-test-slave-vl24 krb5kdc[13000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194571, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:11.925973 13395 init.cc:377] Logged in from keytab as oryx/127.5.83.67@KRBTEST.COM (short username oryx)
I20260923 20:16:11.927373 13395 webserver.cc:460] Webserver started at http://127.5.83.67:41173/ using document root <none> and password file <none>
I20260923 20:16:11.928121 13395 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:11.928241 13395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:11.928654 13395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:11.931565 13395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "4b87c71baf544daabf85c48235a0a479"
format_stamp: "Formatted at 2026-09-23 20:16:11 on dist-test-slave-vl24"
server_key: "23382703e1dcf44a5a0dd7ff6129914a"
server_key_iv: "23221646ed0ba462374e9ff421b5a557"
server_key_version: "encryptionkey@0"
I20260923 20:16:11.932389 13395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "4b87c71baf544daabf85c48235a0a479"
format_stamp: "Formatted at 2026-09-23 20:16:11 on dist-test-slave-vl24"
server_key: "23382703e1dcf44a5a0dd7ff6129914a"
server_key_iv: "23221646ed0ba462374e9ff421b5a557"
server_key_version: "encryptionkey@0"
I20260923 20:16:11.937656 13395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:16:11.941164 13410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:11.942420 13395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:16:11.942629 13395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "4b87c71baf544daabf85c48235a0a479"
format_stamp: "Formatted at 2026-09-23 20:16:11 on dist-test-slave-vl24"
server_key: "23382703e1dcf44a5a0dd7ff6129914a"
server_key_iv: "23221646ed0ba462374e9ff421b5a557"
server_key_version: "encryptionkey@0"
I20260923 20:16:11.942860 13395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:11.971061 13252 heartbeater.cc:503] Master 127.5.83.126:41577 was elected leader, sending a full tablet report...
I20260923 20:16:11.978461 13395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:11.982682 13395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:11.982905 13395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:11.985666 13395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:11.985780 13395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:11.985954 13395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:11.986039 13395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:16:11 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194571, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:16:12.007920 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:11.986058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:50801 (local address 127.5.83.126:41577)
0923 20:16:11.986297 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:11.986306 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:11.987794 (+ 1488us) server_negotiation.cc:407] Connection header received
0923 20:16:11.990018 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:11.990024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:11.990144 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:11.990268 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:11.992268 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:11.993358 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:11.994931 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:11.995331 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:11.995467 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:11.999764 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:11.999770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:11.999776 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:11.999818 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:12.003846 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:12.004942 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:12.004951 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:12.004955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:12.005146 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:12.005868 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:12.005877 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:12.005881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:12.006248 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:12.006499 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.007247 (+ 748us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.007506 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 20:16:12.009450 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:11.986699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41577 (local address 127.5.83.67:50801)
0923 20:16:11.987627 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:11.987666 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:11.989689 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:11.990503 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:11.990512 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:11.991188 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:11.992094 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:11.992113 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:11.993527 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:11.993533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:11.994791 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:11.994802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:11.995556 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:11.995564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:11.995788 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:11.996504 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:11.996564 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:11.999559 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:12.004023 (+ 4464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:12.004035 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:12.004061 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:12.004782 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:12.005324 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:12.005331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:12.005340 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:12.005658 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:12.006424 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:12.006434 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:12.006829 (+ 395us) client_negotiation.cc:769] Sending connection context
0923 20:16:12.007134 (+ 305us) client_negotiation.cc:240] Negotiation successful
0923 20:16:12.007678 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":472,"threads_started":1}
I20260923 20:16:12.031330 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.015864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:35705 (local address 127.5.83.126:41577)
0923 20:16:12.016057 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.016061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.016692 (+ 631us) server_negotiation.cc:407] Connection header received
0923 20:16:12.016989 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.016998 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.017093 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.017208 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:12.018161 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.019386 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.021030 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.021676 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.021825 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.023776 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:12.023782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:12.023785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:12.023818 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:12.027540 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:12.028606 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:12.028615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:12.028619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:12.028835 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:12.029381 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:12.029389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:12.029393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:12.029823 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:12.029945 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.030693 (+ 748us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.030939 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:16:12.031533 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.015896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41577 (local address 127.5.83.67:35705)
0923 20:16:12.016490 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:12.016517 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:12.016787 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:12.017252 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:12.017256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:12.017696 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:12.018016 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.018026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.019542 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.019551 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.020829 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.020852 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.021917 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.021924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.022094 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.022767 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:12.022786 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:12.023542 (+ 756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:12.027709 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:12.027715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:12.027720 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:12.028438 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:12.029019 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:12.029025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:12.029029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:12.029184 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:12.030076 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:12.030083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:12.030379 (+ 296us) client_negotiation.cc:769] Sending connection context
0923 20:16:12.030568 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 20:16:12.030966 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":305,"threads_started":1}
I20260923 20:16:12.039942 13529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:42127 every 8 connection(s)
I20260923 20:16:12.039938 13395 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:42127
I20260923 20:16:12.041265 13395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:16:12.043323 13530 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41577
I20260923 20:16:12.043557 13530 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:12.044183 13530 heartbeater.cc:511] Master 127.5.83.126:41577 requested a full tablet report, sending...
I20260923 20:16:12.045590 13044 ts_manager.cc:194] Registered new tserver with Master: 4b87c71baf544daabf85c48235a0a479 (127.5.83.67:42127)
I20260923 20:16:12.046545 13044 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:50801
I20260923 20:16:12.050384 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 13395
I20260923 20:16:12.050594 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:16:12.051028 5453 external_mini_cluster.cc:1428] Setting key 09120d29cbf6de607027fdd54b03bb60
I20260923 20:16:12.053889 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 20:16:12 dist-test-slave-vl24 krb5kdc[13000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194572, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 20:16:12 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:12 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:16:12.106283 13140 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:12.096095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.5.83.126:41577)
0923 20:16:12.096267 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.096271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.096740 (+ 469us) server_negotiation.cc:407] Connection header received
0923 20:16:12.096894 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.096898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.096967 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.097062 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:12.098370 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.099091 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.100274 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.100784 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.100887 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.106065 (+ 5178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56158: BlockingRecv error: recv got EOF from 127.0.0.1:56158 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 20:16:12 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194572, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:16:12.122598 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.109464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.5.83.126:41577)
0923 20:16:12.109681 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.109687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.109875 (+ 188us) server_negotiation.cc:407] Connection header received
0923 20:16:12.110012 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.110016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.110094 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.110178 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:12.111240 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.112076 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.113352 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.113782 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.113883 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.117671 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:12.117677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:12.117680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:12.117713 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:12.120286 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:12.120963 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:12.120967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:12.120969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:12.121084 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:12.121384 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:12.121387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:12.121389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:12.121677 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:12.121898 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.122103 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.122260 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:16:12.126447 13044 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:16:12.133416 13044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:16:12.169723 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.145937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.5.83.66:40643)
0923 20:16:12.158445 (+ 12508us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.158456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.160681 (+ 2225us) server_negotiation.cc:407] Connection header received
0923 20:16:12.160799 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.160807 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.161104 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.161320 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:12.163178 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.164707 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.167630 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.168573 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.168704 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.168925 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.169221 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.169390 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12295,"thread_start_us":12488,"threads_started":1}
I20260923 20:16:12.169804 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.145644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:40643 (local address 127.0.0.1:40218)
0923 20:16:12.160297 (+ 14653us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:12.160325 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:12.160498 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:12.162454 (+ 1956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:12.162458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:12.162496 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:12.162966 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.162980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.165167 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.165173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.167472 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.167486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.168762 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.168769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.168956 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.169007 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 20:16:12.169162 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 20:16:12.169359 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14475,"mutex_wait_us":25,"thread_start_us":13050,"threads_started":1}
I20260923 20:16:12.170747 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.145500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:36757 (local address 127.0.0.1:52236)
0923 20:16:12.158592 (+ 13092us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:12.158645 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:12.158885 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:12.162254 (+ 3369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:12.162266 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:12.162315 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:12.163377 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.163386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.165245 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.165252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.167775 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.167790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.169950 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.169956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.170145 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.170180 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:16:12.170267 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 20:16:12.170391 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12782,"mutex_wait_us":1,"thread_start_us":15996,"threads_started":1}
I20260923 20:16:12.170768 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.145699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.5.83.65:36757)
0923 20:16:12.160917 (+ 15218us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.160925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.160949 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:16:12.161032 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.161039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.161233 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.161397 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:12.163534 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.165055 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.167969 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.169494 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.169617 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.169806 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.170317 (+ 511us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.170467 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15064,"thread_start_us":15852,"threads_started":1}
I20260923 20:16:12.173524 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.159557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:42127 (local address 127.0.0.1:52950)
0923 20:16:12.162158 (+ 2601us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:12.162182 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:12.162283 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:12.163818 (+ 1535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:12.163823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:12.163853 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:12.164186 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.164197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.166245 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.166255 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.168385 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.168399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.172797 (+ 4398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.172803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.173005 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.173044 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 20:16:12.173132 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 20:16:12.173281 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2478,"thread_start_us":2314,"threads_started":1}
I20260923 20:16:12.174271 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.159912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.5.83.67:42127)
0923 20:16:12.163223 (+ 3311us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.163233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.163266 (+ 33us) server_negotiation.cc:407] Connection header received
0923 20:16:12.163369 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.163376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.163618 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.163826 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:12.164399 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.166057 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.168590 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.171126 (+ 2536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.172753 (+ 1627us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.173008 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.173818 (+ 810us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.173981 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3114,"thread_start_us":2926,"threads_started":1}
I20260923 20:16:12.173997 13321 tablet_service.cc:1467] Processing CreateTablet for tablet 43036afa9362431193ad872cfee68342 (DEFAULT_TABLE table=test-table [id=78a22e389c9143c297a21d7ba311eb2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:12.174386 13182 tablet_service.cc:1467] Processing CreateTablet for tablet 43036afa9362431193ad872cfee68342 (DEFAULT_TABLE table=test-table [id=78a22e389c9143c297a21d7ba311eb2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:12.175797 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43036afa9362431193ad872cfee68342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:12.177577 13321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43036afa9362431193ad872cfee68342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:12.177686 13460 tablet_service.cc:1467] Processing CreateTablet for tablet 43036afa9362431193ad872cfee68342 (DEFAULT_TABLE table=test-table [id=78a22e389c9143c297a21d7ba311eb2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:12.178951 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43036afa9362431193ad872cfee68342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:12.189236 13554 tablet_bootstrap.cc:492] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338: Bootstrap starting.
I20260923 20:16:12.193250 13554 tablet_bootstrap.cc:654] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:12.193830 13556 tablet_bootstrap.cc:492] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479: Bootstrap starting.
I20260923 20:16:12.194638 13554 log.cc:824] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:12.195433 13555 tablet_bootstrap.cc:492] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9: Bootstrap starting.
I20260923 20:16:12.197613 13556 tablet_bootstrap.cc:654] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:12.198695 13555 tablet_bootstrap.cc:654] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:12.198948 13554 tablet_bootstrap.cc:492] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338: No bootstrap required, opened a new log
I20260923 20:16:12.199252 13554 ts_tablet_manager.cc:1397] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 20:16:12.199311 13556 log.cc:824] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:12.199908 13555 log.cc:824] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:12.202296 13556 tablet_bootstrap.cc:492] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479: No bootstrap required, opened a new log
I20260923 20:16:12.202584 13556 ts_tablet_manager.cc:1397] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:16:12.202931 13555 tablet_bootstrap.cc:492] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9: No bootstrap required, opened a new log
I20260923 20:16:12.203179 13555 ts_tablet_manager.cc:1397] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260923 20:16:12.207051 13554 raft_consensus.cc:348] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.207630 13554 raft_consensus.cc:374] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:12.207777 13554 raft_consensus.cc:729] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ea93ba0f394262814ef7f6202be338, State: Initialized, Role: FOLLOWER
I20260923 20:16:12.208618 13554 consensus_queue.cc:260] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.210836 13391 heartbeater.cc:503] Master 127.5.83.126:41577 was elected leader, sending a full tablet report...
I20260923 20:16:12.212944 13554 ts_tablet_manager.cc:1428] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20260923 20:16:12.213384 13556 raft_consensus.cc:348] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.213903 13556 raft_consensus.cc:374] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:12.214031 13556 raft_consensus.cc:729] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b87c71baf544daabf85c48235a0a479, State: Initialized, Role: FOLLOWER
I20260923 20:16:12.214021 13555 raft_consensus.cc:348] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.214463 13555 raft_consensus.cc:374] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:12.214578 13555 raft_consensus.cc:729] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efea12b7cbb41ada3f316cab53be5b9, State: Initialized, Role: FOLLOWER
I20260923 20:16:12.214699 13556 consensus_queue.cc:260] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.215253 13555 consensus_queue.cc:260] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.217103 13555 ts_tablet_manager.cc:1428] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260923 20:16:12.217582 13530 heartbeater.cc:503] Master 127.5.83.126:41577 was elected leader, sending a full tablet report...
I20260923 20:16:12.219282 13556 ts_tablet_manager.cc:1428] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260923 20:16:12.248837 13392 tablet.cc:2369] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:16:12.275339 13561 raft_consensus.cc:482] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:16:12.275686 13561 raft_consensus.cc:504] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.277549 13561 leader_election.cc:288] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efea12b7cbb41ada3f316cab53be5b9 (127.5.83.65:36757), 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643)
I20260923 20:16:12.284976 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.277975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:54681 (local address 127.5.83.65:36757)
0923 20:16:12.278199 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.278206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.278230 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:16:12.278322 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.278330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.278461 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.278570 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:12.279359 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.280406 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.282848 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.283989 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.284120 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.284227 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.284578 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.284733 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 20:16:12.285012 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.278325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:36689 (local address 127.5.83.66:40643)
0923 20:16:12.278549 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.278554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.278946 (+ 392us) server_negotiation.cc:407] Connection header received
0923 20:16:12.279079 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.279083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.279139 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.279208 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:12.279652 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.280678 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.282969 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.284096 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.284199 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.284301 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.284586 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.284725 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 20:16:12.285017 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.277808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:36757 (local address 127.5.83.67:54681)
0923 20:16:12.278024 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:12.278047 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:12.278197 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:12.278656 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:12.278661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:12.278689 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:12.278944 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.278956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.280593 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.280601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.282669 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.282689 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.284129 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.284132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.284340 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.284402 (+ 62us) client_negotiation.cc:769] Sending connection context
0923 20:16:12.284492 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 20:16:12.284686 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":26}
I20260923 20:16:12.285085 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.278083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:40643 (local address 127.5.83.67:36689)
0923 20:16:12.278830 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:12.278846 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:12.278966 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:12.279235 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:12.279238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:12.279256 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:12.279498 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.279504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.280784 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.280789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.282791 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.282802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.284274 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.284280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.284421 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.284454 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:16:12.284522 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 20:16:12.284685 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":1,"thread_start_us":396,"threads_started":1}
I20260923 20:16:12.286154 13202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43036afa9362431193ad872cfee68342" candidate_uuid: "4b87c71baf544daabf85c48235a0a479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efea12b7cbb41ada3f316cab53be5b9" is_pre_election: true
I20260923 20:16:12.286158 13341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43036afa9362431193ad872cfee68342" candidate_uuid: "4b87c71baf544daabf85c48235a0a479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ea93ba0f394262814ef7f6202be338" is_pre_election: true
I20260923 20:16:12.286626 13341 raft_consensus.cc:2454] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b87c71baf544daabf85c48235a0a479 in term 0.
I20260923 20:16:12.286623 13202 raft_consensus.cc:2454] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b87c71baf544daabf85c48235a0a479 in term 0.
I20260923 20:16:12.287266 13414 leader_election.cc:302] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b87c71baf544daabf85c48235a0a479, 50ea93ba0f394262814ef7f6202be338; no voters:
I20260923 20:16:12.287668 13561 raft_consensus.cc:2781] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:16:12.287827 13561 raft_consensus.cc:482] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:16:12.288054 13561 raft_consensus.cc:3037] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:12.290714 13561 raft_consensus.cc:504] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.291365 13561 leader_election.cc:288] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [CANDIDATE]: Term 1 election: Requested vote from peers 3efea12b7cbb41ada3f316cab53be5b9 (127.5.83.65:36757), 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643)
I20260923 20:16:12.291733 13202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43036afa9362431193ad872cfee68342" candidate_uuid: "4b87c71baf544daabf85c48235a0a479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efea12b7cbb41ada3f316cab53be5b9"
I20260923 20:16:12.291883 13341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43036afa9362431193ad872cfee68342" candidate_uuid: "4b87c71baf544daabf85c48235a0a479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ea93ba0f394262814ef7f6202be338"
I20260923 20:16:12.291951 13202 raft_consensus.cc:3037] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:12.292092 13341 raft_consensus.cc:3037] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Advancing to term 1
W20260923 20:16:12.292801 13531 tablet.cc:2369] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:16:12.295257 13202 raft_consensus.cc:2454] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b87c71baf544daabf85c48235a0a479 in term 1.
I20260923 20:16:12.295257 13341 raft_consensus.cc:2454] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b87c71baf544daabf85c48235a0a479 in term 1.
I20260923 20:16:12.295662 13414 leader_election.cc:302] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b87c71baf544daabf85c48235a0a479, 50ea93ba0f394262814ef7f6202be338; no voters:
I20260923 20:16:12.295938 13561 raft_consensus.cc:2781] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:12.296392 13561 raft_consensus.cc:686] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 1 LEADER]: Becoming Leader. State: Replica: 4b87c71baf544daabf85c48235a0a479, State: Running, Role: LEADER
I20260923 20:16:12.296906 13561 consensus_queue.cc:237] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.301553 13045 catalog_manager.cc:5495] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b87c71baf544daabf85c48235a0a479 (127.5.83.67). New cstate: current_term: 1 leader_uuid: "4b87c71baf544daabf85c48235a0a479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } health_report { overall_health: HEALTHY } } }
I20260923 20:16:12.335319 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.329509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.5.83.67:42127)
0923 20:16:12.329696 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.329701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.329717 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:16:12.329908 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.329912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.329991 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.330115 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:16:12.330487 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.331171 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.333095 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.333494 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.333597 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.334180 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:16:12.334728 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:16:12.334834 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.334948 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.335013 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:16:12.343080 13341 raft_consensus.cc:1261] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Refusing update from remote peer 4b87c71baf544daabf85c48235a0a479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:12.343099 13202 raft_consensus.cc:1261] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Refusing update from remote peer 4b87c71baf544daabf85c48235a0a479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:12.344018 13564 consensus_queue.cc:1035] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:12.344368 13561 consensus_queue.cc:1035] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:12.353974 13567 mvcc.cc:205] Tried to move back new op lower bound from 7332636968308035584 to 7332636968131702784. Current Snapshot: MvccSnapshot[applied={T|T < 7332636968308035584}]
I20260923 20:16:12.357535 13568 mvcc.cc:205] Tried to move back new op lower bound from 7332636968308035584 to 7332636968131702784. Current Snapshot: MvccSnapshot[applied={T|T < 7332636968308035584}]
I20260923 20:16:12.359536 13569 mvcc.cc:205] Tried to move back new op lower bound from 7332636968308035584 to 7332636968131702784. Current Snapshot: MvccSnapshot[applied={T|T < 7332636968308035584}]
I20260923 20:16:12.368462 13044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:12.368815 13044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:12.372419 13044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb: Sending DeleteTablet for 3 replicas of tablet 43036afa9362431193ad872cfee68342
I20260923 20:16:12.373711 13460 tablet_service.cc:1514] Processing DeleteTablet for tablet 43036afa9362431193ad872cfee68342 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:12 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:52950
I20260923 20:16:12.373723 13182 tablet_service.cc:1514] Processing DeleteTablet for tablet 43036afa9362431193ad872cfee68342 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:12 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:52236
I20260923 20:16:12.374500 13321 tablet_service.cc:1514] Processing DeleteTablet for tablet 43036afa9362431193ad872cfee68342 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:12 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:40218
I20260923 20:16:12.375038 13577 tablet_replica.cc:330] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9: stopping tablet replica
I20260923 20:16:12.375097 13576 tablet_replica.cc:330] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479: stopping tablet replica
I20260923 20:16:12.375500 13577 raft_consensus.cc:2229] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:12.375542 13576 raft_consensus.cc:2229] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:16:12.376041 13577 raft_consensus.cc:2258] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:12.376307 13576 raft_consensus.cc:2258] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:12.376607 13044 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:16:12.377355 13044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:16:12.378567 13576 ts_tablet_manager.cc:1905] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:12.383667 13577 ts_tablet_manager.cc:1905] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:12.386500 13576 ts_tablet_manager.cc:1918] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:12.386734 13576 log.cc:1196] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/43036afa9362431193ad872cfee68342
I20260923 20:16:12.387396 13576 ts_tablet_manager.cc:1939] T 43036afa9362431193ad872cfee68342 P 4b87c71baf544daabf85c48235a0a479: Deleting consensus metadata
I20260923 20:16:12.387336 13320 tablet_service.cc:1467] Processing CreateTablet for tablet a760c8001aad46e9bc20d742dbd24569 (DEFAULT_TABLE table=test-table [id=f3fd0db5d0e141bb85509189307e85d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:12.388053 13320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a760c8001aad46e9bc20d742dbd24569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:12.392602 13578 tablet_replica.cc:330] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338: stopping tablet replica
I20260923 20:16:12.392894 13460 tablet_service.cc:1467] Processing CreateTablet for tablet a760c8001aad46e9bc20d742dbd24569 (DEFAULT_TABLE table=test-table [id=f3fd0db5d0e141bb85509189307e85d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:12.393055 13578 raft_consensus.cc:2229] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:12.393280 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a760c8001aad46e9bc20d742dbd24569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:12.393514 13578 raft_consensus.cc:2258] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:12.395742 13578 ts_tablet_manager.cc:1905] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:12.396126 13577 ts_tablet_manager.cc:1918] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:12.396284 13577 log.cc:1196] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/43036afa9362431193ad872cfee68342
I20260923 20:16:12.396903 13577 ts_tablet_manager.cc:1939] T 43036afa9362431193ad872cfee68342 P 3efea12b7cbb41ada3f316cab53be5b9: Deleting consensus metadata
I20260923 20:16:12.397568 13032 catalog_manager.cc:4841] TS 4b87c71baf544daabf85c48235a0a479 (127.5.83.67:42127): tablet 43036afa9362431193ad872cfee68342 (table test-table [id=78a22e389c9143c297a21d7ba311eb2f]) successfully deleted
I20260923 20:16:12.398676 13182 tablet_service.cc:1467] Processing CreateTablet for tablet a760c8001aad46e9bc20d742dbd24569 (DEFAULT_TABLE table=test-table [id=f3fd0db5d0e141bb85509189307e85d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:12.399114 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a760c8001aad46e9bc20d742dbd24569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:12.401980 13556 tablet_bootstrap.cc:492] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479: Bootstrap starting.
I20260923 20:16:12.402350 13031 catalog_manager.cc:4841] TS 3efea12b7cbb41ada3f316cab53be5b9 (127.5.83.65:36757): tablet 43036afa9362431193ad872cfee68342 (table test-table [id=78a22e389c9143c297a21d7ba311eb2f]) successfully deleted
I20260923 20:16:12.402498 13578 ts_tablet_manager.cc:1918] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:12.402782 13578 log.cc:1196] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/43036afa9362431193ad872cfee68342
I20260923 20:16:12.403326 13554 tablet_bootstrap.cc:492] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338: Bootstrap starting.
I20260923 20:16:12.403581 13556 tablet_bootstrap.cc:654] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:12.404897 13554 tablet_bootstrap.cc:654] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:12.409562 13578 ts_tablet_manager.cc:1939] T 43036afa9362431193ad872cfee68342 P 50ea93ba0f394262814ef7f6202be338: Deleting consensus metadata
I20260923 20:16:12.411324 13555 tablet_bootstrap.cc:492] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9: Bootstrap starting.
I20260923 20:16:12.413363 13555 tablet_bootstrap.cc:654] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:12.413496 13032 catalog_manager.cc:4841] TS 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643): tablet 43036afa9362431193ad872cfee68342 (table test-table [id=78a22e389c9143c297a21d7ba311eb2f]) successfully deleted
I20260923 20:16:12.418102 13554 tablet_bootstrap.cc:492] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338: No bootstrap required, opened a new log
I20260923 20:16:12.418320 13554 ts_tablet_manager.cc:1397] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20260923 20:16:12.419234 13554 raft_consensus.cc:348] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.419612 13554 raft_consensus.cc:374] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:12.419800 13554 raft_consensus.cc:729] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ea93ba0f394262814ef7f6202be338, State: Initialized, Role: FOLLOWER
I20260923 20:16:12.420286 13554 consensus_queue.cc:260] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.421653 13554 ts_tablet_manager.cc:1428] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 20:16:12.424729 13555 tablet_bootstrap.cc:492] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9: No bootstrap required, opened a new log
I20260923 20:16:12.424943 13555 ts_tablet_manager.cc:1397] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.002s
I20260923 20:16:12.425872 13555 raft_consensus.cc:348] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.426172 13555 raft_consensus.cc:374] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:12.426321 13555 raft_consensus.cc:729] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efea12b7cbb41ada3f316cab53be5b9, State: Initialized, Role: FOLLOWER
I20260923 20:16:12.426652 13555 consensus_queue.cc:260] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.427306 13555 ts_tablet_manager.cc:1428] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:16:12.429225 13556 tablet_bootstrap.cc:492] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479: No bootstrap required, opened a new log
I20260923 20:16:12.429440 13556 ts_tablet_manager.cc:1397] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.014s
I20260923 20:16:12.430459 13556 raft_consensus.cc:348] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.430706 13556 raft_consensus.cc:374] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:12.430811 13556 raft_consensus.cc:729] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b87c71baf544daabf85c48235a0a479, State: Initialized, Role: FOLLOWER
I20260923 20:16:12.431097 13556 consensus_queue.cc:260] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.431713 13556 ts_tablet_manager.cc:1428] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260923 20:16:12.460906 13253 tablet.cc:2369] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:16:12.534284 13562 raft_consensus.cc:482] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:16:12.534548 13562 raft_consensus.cc:504] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.536136 13562 leader_election.cc:288] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643), 4b87c71baf544daabf85c48235a0a479 (127.5.83.67:42127)
I20260923 20:16:12.543483 13551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.536521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:51583 (local address 127.5.83.66:40643)
0923 20:16:12.536779 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.536787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.537342 (+ 555us) server_negotiation.cc:407] Connection header received
0923 20:16:12.537625 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.537631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.537724 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.537786 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:12.538610 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.539730 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.541664 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.542428 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.542574 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.542686 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.543039 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.543187 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 20:16:12.543467 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.536415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:40643 (local address 127.5.83.65:51583)
0923 20:16:12.537230 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:12.537249 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:12.537461 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:12.537999 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:12.538004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:12.538064 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:12.538417 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.538429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.539871 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.539876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.541508 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.541520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.542668 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.542675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.542863 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.542898 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:16:12.542995 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:16:12.543122 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":516,"threads_started":1}
I20260923 20:16:12.544575 13341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a760c8001aad46e9bc20d742dbd24569" candidate_uuid: "3efea12b7cbb41ada3f316cab53be5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ea93ba0f394262814ef7f6202be338" is_pre_election: true
I20260923 20:16:12.544821 13341 raft_consensus.cc:2454] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efea12b7cbb41ada3f316cab53be5b9 in term 0.
I20260923 20:16:12.545195 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.537316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:42127 (local address 127.5.83.65:37233)
0923 20:16:12.538043 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:12.538064 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:12.538249 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:12.538798 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:12.538802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:12.538827 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:12.539227 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.539244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.540573 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.540578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.542666 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.542681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.544383 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.544389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.544630 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.544653 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:16:12.544751 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 20:16:12.544869 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":1651,"threads_started":1}
I20260923 20:16:12.545213 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.537423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:37233 (local address 127.5.83.67:42127)
0923 20:16:12.537609 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.537615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.538140 (+ 525us) server_negotiation.cc:407] Connection header received
0923 20:16:12.538441 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.538445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.538518 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.538592 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:12.539400 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.540370 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.542836 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.544187 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.544333 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.544430 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.544791 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.544915 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:16:12.545433 13135 leader_election.cc:302] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efea12b7cbb41ada3f316cab53be5b9, 50ea93ba0f394262814ef7f6202be338; no voters:
I20260923 20:16:12.545876 13562 raft_consensus.cc:2781] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:16:12.546010 13562 raft_consensus.cc:482] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:16:12.546131 13562 raft_consensus.cc:3037] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:12.546473 13485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a760c8001aad46e9bc20d742dbd24569" candidate_uuid: "3efea12b7cbb41ada3f316cab53be5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b87c71baf544daabf85c48235a0a479" is_pre_election: true
I20260923 20:16:12.546792 13485 raft_consensus.cc:2454] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efea12b7cbb41ada3f316cab53be5b9 in term 0.
I20260923 20:16:12.548708 13562 raft_consensus.cc:504] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.549270 13562 leader_election.cc:288] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [CANDIDATE]: Term 1 election: Requested vote from peers 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643), 4b87c71baf544daabf85c48235a0a479 (127.5.83.67:42127)
I20260923 20:16:12.549777 13485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a760c8001aad46e9bc20d742dbd24569" candidate_uuid: "3efea12b7cbb41ada3f316cab53be5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b87c71baf544daabf85c48235a0a479"
I20260923 20:16:12.549798 13341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a760c8001aad46e9bc20d742dbd24569" candidate_uuid: "3efea12b7cbb41ada3f316cab53be5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ea93ba0f394262814ef7f6202be338"
I20260923 20:16:12.549980 13485 raft_consensus.cc:3037] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:12.550017 13341 raft_consensus.cc:3037] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:12.552409 13485 raft_consensus.cc:2454] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efea12b7cbb41ada3f316cab53be5b9 in term 1.
I20260923 20:16:12.552811 13135 leader_election.cc:302] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efea12b7cbb41ada3f316cab53be5b9, 4b87c71baf544daabf85c48235a0a479; no voters:
I20260923 20:16:12.553026 13341 raft_consensus.cc:2454] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efea12b7cbb41ada3f316cab53be5b9 in term 1.
I20260923 20:16:12.553064 13562 raft_consensus.cc:2781] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:12.553887 13562 raft_consensus.cc:686] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 LEADER]: Becoming Leader. State: Replica: 3efea12b7cbb41ada3f316cab53be5b9, State: Running, Role: LEADER
I20260923 20:16:12.554283 13562 consensus_queue.cc:237] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.557925 13045 catalog_manager.cc:5495] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efea12b7cbb41ada3f316cab53be5b9 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "3efea12b7cbb41ada3f316cab53be5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } health_report { overall_health: UNKNOWN } } }
Sep 23 20:16:12 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194570, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.126@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:16:12.638626 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.619585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41577 (local address 127.0.0.1:56184)
0923 20:16:12.620265 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:12.620287 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:12.620460 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:12.620883 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:12.620891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:12.621949 (+ 1058us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:12.622311 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.622327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.623825 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.623833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.625688 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.625704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.626711 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.626718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.626944 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.627757 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:12.627778 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:12.630736 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:12.635034 (+ 4298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:12.635059 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:12.635068 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:12.635676 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:12.636573 (+ 897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:12.636578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:12.636582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:12.636776 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:12.637460 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:12.637469 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:12.637662 (+ 193us) client_negotiation.cc:769] Sending connection context
0923 20:16:12.637832 (+ 170us) client_negotiation.cc:240] Negotiation successful
0923 20:16:12.638145 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":353,"threads_started":1}
I20260923 20:16:12.638981 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.619544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56184 (local address 127.5.83.126:41577)
0923 20:16:12.619758 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.619764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.620399 (+ 635us) server_negotiation.cc:407] Connection header received
0923 20:16:12.620602 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.620609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.620690 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.620806 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:12.622482 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.623682 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.625847 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.626503 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.626651 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.630964 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:12.630972 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:12.630980 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:12.631020 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:12.634843 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:12.635821 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:12.635828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:12.635833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:12.636007 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:12.636913 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:12.636928 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:12.636933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:12.637316 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:12.637472 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.638035 (+ 563us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.638287 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:16:12.645448 13045 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:56184:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 20:16:12.646816 13045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:16:12.654995 13182 tablet_service.cc:1467] Processing CreateTablet for tablet d64c7906b0184ebe8920d783c4a26d47 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97c7fa0ac9b44d768692286e694fc3fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:16:12.655345 13320 tablet_service.cc:1467] Processing CreateTablet for tablet d64c7906b0184ebe8920d783c4a26d47 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97c7fa0ac9b44d768692286e694fc3fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:16:12.655387 13460 tablet_service.cc:1467] Processing CreateTablet for tablet d64c7906b0184ebe8920d783c4a26d47 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97c7fa0ac9b44d768692286e694fc3fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 20:16:12.655647 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64c7906b0184ebe8920d783c4a26d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:12.655864 13320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64c7906b0184ebe8920d783c4a26d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:12.655941 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d64c7906b0184ebe8920d783c4a26d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:12.663339 13555 tablet_bootstrap.cc:492] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9: Bootstrap starting.
I20260923 20:16:12.664002 13554 tablet_bootstrap.cc:492] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338: Bootstrap starting.
I20260923 20:16:12.665449 13555 tablet_bootstrap.cc:654] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:12.665720 13554 tablet_bootstrap.cc:654] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:12.669117 13555 tablet_bootstrap.cc:492] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9: No bootstrap required, opened a new log
I20260923 20:16:12.669430 13555 ts_tablet_manager.cc:1397] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260923 20:16:12.671695 13555 raft_consensus.cc:348] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.672117 13555 raft_consensus.cc:374] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:12.672247 13555 raft_consensus.cc:729] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efea12b7cbb41ada3f316cab53be5b9, State: Initialized, Role: FOLLOWER
I20260923 20:16:12.672619 13555 consensus_queue.cc:260] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.674558 13562 tablet_replica.cc:439] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } } }
I20260923 20:16:12.674882 13562 tablet_replica.cc:442] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:16:12.674775 13555 ts_tablet_manager.cc:1428] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260923 20:16:12.675423 13556 tablet_bootstrap.cc:492] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: Bootstrap starting.
I20260923 20:16:12.676574 13554 tablet_bootstrap.cc:492] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338: No bootstrap required, opened a new log
I20260923 20:16:12.676762 13554 ts_tablet_manager.cc:1397] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260923 20:16:12.677660 13556 tablet_bootstrap.cc:654] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:12.677543 13554 raft_consensus.cc:348] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.677795 13554 raft_consensus.cc:374] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:12.677871 13554 raft_consensus.cc:729] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ea93ba0f394262814ef7f6202be338, State: Initialized, Role: FOLLOWER
I20260923 20:16:12.678162 13554 consensus_queue.cc:260] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.678786 13554 ts_tablet_manager.cc:1428] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 20:16:12.678676 13560 tablet_replica.cc:439] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } } }
I20260923 20:16:12.679075 13560 tablet_replica.cc:442] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:16:12.690296 13556 tablet_bootstrap.cc:492] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: No bootstrap required, opened a new log
I20260923 20:16:12.690508 13556 ts_tablet_manager.cc:1397] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260923 20:16:12.691350 13556 raft_consensus.cc:348] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.691638 13556 raft_consensus.cc:374] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:12.691747 13556 raft_consensus.cc:729] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b87c71baf544daabf85c48235a0a479, State: Initialized, Role: FOLLOWER
I20260923 20:16:12.692078 13556 consensus_queue.cc:260] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.692492 13564 tablet_replica.cc:439] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } } }
I20260923 20:16:12.692734 13556 ts_tablet_manager.cc:1428] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 20:16:12.692900 13564 tablet_replica.cc:442] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:16:12.884521 13564 raft_consensus.cc:482] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:16:12.884792 13564 raft_consensus.cc:504] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.885380 13564 leader_election.cc:288] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efea12b7cbb41ada3f316cab53be5b9 (127.5.83.65:36757), 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643)
I20260923 20:16:12.885946 13341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64c7906b0184ebe8920d783c4a26d47" candidate_uuid: "4b87c71baf544daabf85c48235a0a479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ea93ba0f394262814ef7f6202be338" is_pre_election: true
I20260923 20:16:12.886014 13202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64c7906b0184ebe8920d783c4a26d47" candidate_uuid: "4b87c71baf544daabf85c48235a0a479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efea12b7cbb41ada3f316cab53be5b9" is_pre_election: true
I20260923 20:16:12.886220 13341 raft_consensus.cc:2454] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b87c71baf544daabf85c48235a0a479 in term 0.
I20260923 20:16:12.886315 13202 raft_consensus.cc:2454] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b87c71baf544daabf85c48235a0a479 in term 0.
I20260923 20:16:12.886698 13414 leader_election.cc:302] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b87c71baf544daabf85c48235a0a479, 50ea93ba0f394262814ef7f6202be338; no voters:
I20260923 20:16:12.886981 13564 raft_consensus.cc:2781] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:16:12.887064 13564 raft_consensus.cc:482] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:16:12.887197 13564 raft_consensus.cc:3037] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:12.889394 13564 raft_consensus.cc:504] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.889907 13564 leader_election.cc:288] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [CANDIDATE]: Term 1 election: Requested vote from peers 3efea12b7cbb41ada3f316cab53be5b9 (127.5.83.65:36757), 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643)
I20260923 20:16:12.890460 13341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64c7906b0184ebe8920d783c4a26d47" candidate_uuid: "4b87c71baf544daabf85c48235a0a479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ea93ba0f394262814ef7f6202be338"
I20260923 20:16:12.890482 13202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d64c7906b0184ebe8920d783c4a26d47" candidate_uuid: "4b87c71baf544daabf85c48235a0a479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efea12b7cbb41ada3f316cab53be5b9"
I20260923 20:16:12.890668 13202 raft_consensus.cc:3037] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:12.890687 13341 raft_consensus.cc:3037] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:12.893476 13341 raft_consensus.cc:2454] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b87c71baf544daabf85c48235a0a479 in term 1.
I20260923 20:16:12.893491 13202 raft_consensus.cc:2454] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b87c71baf544daabf85c48235a0a479 in term 1.
I20260923 20:16:12.893888 13414 leader_election.cc:302] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b87c71baf544daabf85c48235a0a479, 50ea93ba0f394262814ef7f6202be338; no voters:
I20260923 20:16:12.894174 13564 raft_consensus.cc:2781] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:12.894511 13564 raft_consensus.cc:686] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [term 1 LEADER]: Becoming Leader. State: Replica: 4b87c71baf544daabf85c48235a0a479, State: Running, Role: LEADER
I20260923 20:16:12.894745 13564 consensus_queue.cc:237] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:12.896278 13595 tablet_replica.cc:439] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: TxnStatusTablet state changed. Reason: New leader 4b87c71baf544daabf85c48235a0a479. Latest consensus state: current_term: 1 leader_uuid: "4b87c71baf544daabf85c48235a0a479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } } }
I20260923 20:16:12.896618 13595 tablet_replica.cc:442] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:16:12.897203 13597 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 20:16:12.897408 13597 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 20:16:12.897706 13044 catalog_manager.cc:5495] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b87c71baf544daabf85c48235a0a479 (127.5.83.67). New cstate: current_term: 1 leader_uuid: "4b87c71baf544daabf85c48235a0a479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } health_report { overall_health: HEALTHY } } }
I20260923 20:16:12.923419 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.917659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.5.83.67:42127)
0923 20:16:12.917832 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.917837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.917965 (+ 128us) server_negotiation.cc:407] Connection header received
0923 20:16:12.918223 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.918226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.918310 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.918383 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:16:12.918831 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.919537 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.921665 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.922095 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.922253 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.922667 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:16:12.922765 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:16:12.922866 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.923020 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.923110 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:16:12.923468 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.917620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:42127 (local address 127.0.0.1:52968)
0923 20:16:12.917866 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:12.917890 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:12.918051 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:12.918439 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:12.918445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:12.918460 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:16:12.918692 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.918705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.919670 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.919676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.921529 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.921544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.922278 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.922283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.922430 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.922483 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:16:12.922867 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:16:12.922898 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 20:16:12.922987 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 20:16:12.923171 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 20:16:12.927060 13202 raft_consensus.cc:1261] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Refusing update from remote peer 4b87c71baf544daabf85c48235a0a479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:12.927147 13341 raft_consensus.cc:1261] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Refusing update from remote peer 4b87c71baf544daabf85c48235a0a479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:12.927683 13595 consensus_queue.cc:1035] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:12.927986 13564 consensus_queue.cc:1035] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:12.934012 13560 tablet_replica.cc:439] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338: TxnStatusTablet state changed. Reason: New leader 4b87c71baf544daabf85c48235a0a479. Latest consensus state: current_term: 1 leader_uuid: "4b87c71baf544daabf85c48235a0a479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } } }
I20260923 20:16:12.934371 13560 tablet_replica.cc:442] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:16:12.939371 13595 tablet_replica.cc:439] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b87c71baf544daabf85c48235a0a479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } } }
I20260923 20:16:12.939682 13595 tablet_replica.cc:442] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:16:12.941093 13560 tablet_replica.cc:439] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b87c71baf544daabf85c48235a0a479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } } }
I20260923 20:16:12.941339 13560 tablet_replica.cc:442] T d64c7906b0184ebe8920d783c4a26d47 P 50ea93ba0f394262814ef7f6202be338: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:16:12.945489 13562 tablet_replica.cc:439] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9: TxnStatusTablet state changed. Reason: New leader 4b87c71baf544daabf85c48235a0a479. Latest consensus state: current_term: 1 leader_uuid: "4b87c71baf544daabf85c48235a0a479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } } }
I20260923 20:16:12.945775 13562 tablet_replica.cc:442] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:16:12.948024 13564 tablet_replica.cc:439] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b87c71baf544daabf85c48235a0a479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } } }
I20260923 20:16:12.948359 13564 tablet_replica.cc:442] T d64c7906b0184ebe8920d783c4a26d47 P 4b87c71baf544daabf85c48235a0a479: This TxnStatusTablet replica's current role is: LEADER
I20260923 20:16:12.953063 13562 tablet_replica.cc:439] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b87c71baf544daabf85c48235a0a479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } } }
I20260923 20:16:12.953377 13562 tablet_replica.cc:442] T d64c7906b0184ebe8920d783c4a26d47 P 3efea12b7cbb41ada3f316cab53be5b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 20:16:12.957005 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.945151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.5.83.65:36757)
0923 20:16:12.947335 (+ 2184us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.947342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.947358 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:16:12.947452 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.947456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.947557 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.947735 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:16:12.948485 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.949746 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.954296 (+ 4550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.954731 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.954867 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.955490 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:16:12.956244 (+ 754us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:16:12.956378 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.956562 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.956664 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2055,"thread_start_us":1952,"threads_started":1}
I20260923 20:16:12.980312 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.971396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:51671 (local address 127.5.83.67:42127)
0923 20:16:12.971618 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:12.971624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:12.972289 (+ 665us) server_negotiation.cc:407] Connection header received
0923 20:16:12.972489 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:12.972495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:12.972615 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:12.972718 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:16:12.973770 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.975109 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.977694 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.978503 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.978620 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.979359 (+ 739us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:16:12.979555 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:16:12.979625 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:12.979860 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 20:16:12.979953 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:16:12.981122 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:12.971455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:42127 (local address 127.5.83.65:51671)
0923 20:16:12.972177 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:12.972197 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:12.972337 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:12.972967 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:12.972972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:12.972987 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:16:12.973556 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.973572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.975349 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.975359 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.977515 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:12.977532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:12.978834 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:12.978842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:12.979041 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:12.979121 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:16:12.979715 (+ 594us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:16:12.979727 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 20:16:12.980590 (+ 863us) client_negotiation.cc:240] Negotiation successful
0923 20:16:12.980731 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":380,"threads_started":1}
I20260923 20:16:12.991755 13341 raft_consensus.cc:1261] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Refusing update from remote peer 3efea12b7cbb41ada3f316cab53be5b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:12.991837 13485 raft_consensus.cc:1261] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Refusing update from remote peer 3efea12b7cbb41ada3f316cab53be5b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:12.992501 13584 consensus_queue.cc:1035] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:12.992973 13562 consensus_queue.cc:1035] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:13.035127 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:13.025094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:54561 (local address 127.5.83.65:36757)
0923 20:16:13.025273 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:13.025280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:13.029452 (+ 4172us) server_negotiation.cc:407] Connection header received
0923 20:16:13.029678 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:13.029686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:13.029780 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:13.029874 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:16:13.030493 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:13.031211 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:13.033101 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:13.033721 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:13.033872 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:13.034523 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:16:13.034618 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:16:13.034688 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:13.034829 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 20:16:13.034891 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:16:13.035444 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:13.025088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:36757 (local address 127.5.83.67:54561)
0923 20:16:13.029354 (+ 4266us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:13.029376 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:13.029533 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:13.029907 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:13.029912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:13.029929 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 20:16:13.030344 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:13.030353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:13.031345 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:13.031351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:13.032959 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:13.032972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:13.034040 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:13.034048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:13.034281 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:13.034359 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 20:16:13.034713 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 20:16:13.034725 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 20:16:13.034820 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 20:16:13.034987 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4026,"thread_start_us":3855,"threads_started":1}
I20260923 20:16:13.084802 13044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:13.085117 13044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:13.088975 13044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb: Sending DeleteTablet for 3 replicas of tablet a760c8001aad46e9bc20d742dbd24569
I20260923 20:16:13.090276 13320 tablet_service.cc:1514] Processing DeleteTablet for tablet a760c8001aad46e9bc20d742dbd24569 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:13 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:40218
I20260923 20:16:13.090732 13460 tablet_service.cc:1514] Processing DeleteTablet for tablet a760c8001aad46e9bc20d742dbd24569 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:13 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:52950
I20260923 20:16:13.091355 13182 tablet_service.cc:1514] Processing DeleteTablet for tablet a760c8001aad46e9bc20d742dbd24569 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:13 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:52236
I20260923 20:16:13.091993 13623 tablet_replica.cc:330] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9: stopping tablet replica
I20260923 20:16:13.092190 13623 raft_consensus.cc:2229] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:16:13.092798 13623 raft_consensus.cc:2258] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:13.094022 13623 ts_tablet_manager.cc:1905] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:13.102385 13623 ts_tablet_manager.cc:1918] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:16:13.102646 13623 log.cc:1196] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/a760c8001aad46e9bc20d742dbd24569
I20260923 20:16:13.103282 13623 ts_tablet_manager.cc:1939] T a760c8001aad46e9bc20d742dbd24569 P 3efea12b7cbb41ada3f316cab53be5b9: Deleting consensus metadata
I20260923 20:16:13.104502 13031 catalog_manager.cc:4841] TS 3efea12b7cbb41ada3f316cab53be5b9 (127.5.83.65:36757): tablet a760c8001aad46e9bc20d742dbd24569 (table test-table [id=f3fd0db5d0e141bb85509189307e85d6]) successfully deleted
I20260923 20:16:13.110466 13621 tablet_replica.cc:330] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338: stopping tablet replica
I20260923 20:16:13.110702 13621 raft_consensus.cc:2229] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:13.111112 13621 raft_consensus.cc:2258] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:13.112478 13621 ts_tablet_manager.cc:1905] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:13.112663 13622 tablet_replica.cc:330] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479: stopping tablet replica
I20260923 20:16:13.112949 13622 raft_consensus.cc:2229] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:13.113271 13622 raft_consensus.cc:2258] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:13.115953 13622 ts_tablet_manager.cc:1905] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:13.129855 13622 ts_tablet_manager.cc:1918] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:16:13.133028 13622 log.cc:1196] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/a760c8001aad46e9bc20d742dbd24569
I20260923 20:16:13.133661 13622 ts_tablet_manager.cc:1939] T a760c8001aad46e9bc20d742dbd24569 P 4b87c71baf544daabf85c48235a0a479: Deleting consensus metadata
I20260923 20:16:13.135020 13032 catalog_manager.cc:4841] TS 4b87c71baf544daabf85c48235a0a479 (127.5.83.67:42127): tablet a760c8001aad46e9bc20d742dbd24569 (table test-table [id=f3fd0db5d0e141bb85509189307e85d6]) successfully deleted
I20260923 20:16:13.136689 13621 ts_tablet_manager.cc:1918] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:16:13.136909 13621 log.cc:1196] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/a760c8001aad46e9bc20d742dbd24569
I20260923 20:16:13.137485 13621 ts_tablet_manager.cc:1939] T a760c8001aad46e9bc20d742dbd24569 P 50ea93ba0f394262814ef7f6202be338: Deleting consensus metadata
I20260923 20:16:13.138705 13032 catalog_manager.cc:4841] TS 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643): tablet a760c8001aad46e9bc20d742dbd24569 (table test-table [id=f3fd0db5d0e141bb85509189307e85d6]) successfully deleted
Sep 23 20:16:13 dist-test-slave-vl24 krb5kdc[13000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 20:16:13 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:13 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:16:13.198881 13630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:13.173433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.5.83.126:41577)
0923 20:16:13.185323 (+ 11890us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:13.185329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:13.185350 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:16:13.185417 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:13.185424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:13.185515 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:13.185649 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:13.187573 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:13.188777 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:13.190846 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:13.191512 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:13.191655 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:13.198537 (+ 6882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56194: BlockingRecv error: recv got EOF from 127.0.0.1:56194 (error 108)
Metrics: {"server-negotiator.queue_time_us":11747,"thread_start_us":11640,"threads_started":1}
Sep 23 20:16:13 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM
I20260923 20:16:13.222810 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:13.202778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.5.83.126:41577)
0923 20:16:13.202947 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:13.202952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:13.202973 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:16:13.203303 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:13.203309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:13.203408 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:13.203519 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:13.205232 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:13.206436 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:13.208521 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:13.209220 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:13.209345 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:13.214761 (+ 5416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:13.214771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:13.214777 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:13.214825 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:13.218737 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:13.219668 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:13.219674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:13.219678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:13.219840 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:13.220293 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:13.220298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:13.220302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:13.220699 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:13.220968 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:13.222174 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0923 20:16:13.222430 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:16:13.227136 13044 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:16:13.227890 13044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:16:13.236716 13182 tablet_service.cc:1467] Processing CreateTablet for tablet b96b30d4f3b1411f90aaeb016792fb66 (DEFAULT_TABLE table=test-table [id=6437c4c9d21247e883b16ed74cfc2522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:13.236972 13460 tablet_service.cc:1467] Processing CreateTablet for tablet b96b30d4f3b1411f90aaeb016792fb66 (DEFAULT_TABLE table=test-table [id=6437c4c9d21247e883b16ed74cfc2522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:13.237193 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96b30d4f3b1411f90aaeb016792fb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:13.237473 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96b30d4f3b1411f90aaeb016792fb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:13.239241 13320 tablet_service.cc:1467] Processing CreateTablet for tablet b96b30d4f3b1411f90aaeb016792fb66 (DEFAULT_TABLE table=test-table [id=6437c4c9d21247e883b16ed74cfc2522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:13.239602 13320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96b30d4f3b1411f90aaeb016792fb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:13.251950 13637 tablet_bootstrap.cc:492] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9: Bootstrap starting.
I20260923 20:16:13.252671 13639 tablet_bootstrap.cc:492] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479: Bootstrap starting.
I20260923 20:16:13.252983 13638 tablet_bootstrap.cc:492] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338: Bootstrap starting.
I20260923 20:16:13.255167 13637 tablet_bootstrap.cc:654] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:13.255789 13639 tablet_bootstrap.cc:654] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:13.262627 13638 tablet_bootstrap.cc:654] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:13.263885 13637 tablet_bootstrap.cc:492] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9: No bootstrap required, opened a new log
I20260923 20:16:13.264132 13637 ts_tablet_manager.cc:1397] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260923 20:16:13.265187 13637 raft_consensus.cc:348] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.265626 13637 raft_consensus.cc:374] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:13.265782 13637 raft_consensus.cc:729] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efea12b7cbb41ada3f316cab53be5b9, State: Initialized, Role: FOLLOWER
I20260923 20:16:13.266161 13637 consensus_queue.cc:260] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.266949 13637 ts_tablet_manager.cc:1428] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 20:16:13.274112 13638 tablet_bootstrap.cc:492] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338: No bootstrap required, opened a new log
I20260923 20:16:13.274294 13638 ts_tablet_manager.cc:1397] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260923 20:16:13.275166 13639 tablet_bootstrap.cc:492] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479: No bootstrap required, opened a new log
I20260923 20:16:13.275318 13639 ts_tablet_manager.cc:1397] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.006s
I20260923 20:16:13.275141 13638 raft_consensus.cc:348] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.275396 13638 raft_consensus.cc:374] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:13.275487 13638 raft_consensus.cc:729] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ea93ba0f394262814ef7f6202be338, State: Initialized, Role: FOLLOWER
I20260923 20:16:13.275738 13638 consensus_queue.cc:260] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.276281 13638 ts_tablet_manager.cc:1428] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 20:16:13.276315 13639 raft_consensus.cc:348] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.280745 13639 raft_consensus.cc:374] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:13.280864 13639 raft_consensus.cc:729] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b87c71baf544daabf85c48235a0a479, State: Initialized, Role: FOLLOWER
I20260923 20:16:13.281131 13639 consensus_queue.cc:260] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.283741 13639 ts_tablet_manager.cc:1428] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20260923 20:16:13.445958 13616 raft_consensus.cc:482] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:16:13.446223 13616 raft_consensus.cc:504] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.447172 13616 leader_election.cc:288] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643), 4b87c71baf544daabf85c48235a0a479 (127.5.83.67:42127)
I20260923 20:16:13.447760 13485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96b30d4f3b1411f90aaeb016792fb66" candidate_uuid: "3efea12b7cbb41ada3f316cab53be5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b87c71baf544daabf85c48235a0a479" is_pre_election: true
I20260923 20:16:13.447741 13340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96b30d4f3b1411f90aaeb016792fb66" candidate_uuid: "3efea12b7cbb41ada3f316cab53be5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ea93ba0f394262814ef7f6202be338" is_pre_election: true
I20260923 20:16:13.448009 13340 raft_consensus.cc:2454] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efea12b7cbb41ada3f316cab53be5b9 in term 0.
I20260923 20:16:13.448009 13485 raft_consensus.cc:2454] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efea12b7cbb41ada3f316cab53be5b9 in term 0.
I20260923 20:16:13.448596 13135 leader_election.cc:302] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efea12b7cbb41ada3f316cab53be5b9, 4b87c71baf544daabf85c48235a0a479; no voters:
I20260923 20:16:13.448892 13616 raft_consensus.cc:2781] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:16:13.449034 13616 raft_consensus.cc:482] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:16:13.449150 13616 raft_consensus.cc:3037] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:13.451591 13616 raft_consensus.cc:504] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.452088 13616 leader_election.cc:288] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [CANDIDATE]: Term 1 election: Requested vote from peers 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643), 4b87c71baf544daabf85c48235a0a479 (127.5.83.67:42127)
I20260923 20:16:13.452638 13485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96b30d4f3b1411f90aaeb016792fb66" candidate_uuid: "3efea12b7cbb41ada3f316cab53be5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b87c71baf544daabf85c48235a0a479"
I20260923 20:16:13.452666 13340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96b30d4f3b1411f90aaeb016792fb66" candidate_uuid: "3efea12b7cbb41ada3f316cab53be5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ea93ba0f394262814ef7f6202be338"
I20260923 20:16:13.452826 13485 raft_consensus.cc:3037] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:13.452894 13340 raft_consensus.cc:3037] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:13.454793 13485 raft_consensus.cc:2454] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efea12b7cbb41ada3f316cab53be5b9 in term 1.
I20260923 20:16:13.455219 13135 leader_election.cc:302] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efea12b7cbb41ada3f316cab53be5b9, 4b87c71baf544daabf85c48235a0a479; no voters:
I20260923 20:16:13.455513 13616 raft_consensus.cc:2781] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:13.455674 13616 raft_consensus.cc:686] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 LEADER]: Becoming Leader. State: Replica: 3efea12b7cbb41ada3f316cab53be5b9, State: Running, Role: LEADER
I20260923 20:16:13.456002 13340 raft_consensus.cc:2454] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efea12b7cbb41ada3f316cab53be5b9 in term 1.
I20260923 20:16:13.456000 13616 consensus_queue.cc:237] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.459072 13044 catalog_manager.cc:5495] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efea12b7cbb41ada3f316cab53be5b9 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "3efea12b7cbb41ada3f316cab53be5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } health_report { overall_health: UNKNOWN } } }
I20260923 20:16:13.491241 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:13.485992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.5.83.65:36757)
0923 20:16:13.486165 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:13.486171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:13.486334 (+ 163us) server_negotiation.cc:407] Connection header received
0923 20:16:13.486570 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:13.486574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:13.486660 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:13.486759 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:16:13.487146 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:13.487856 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:13.489334 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:13.489874 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:13.490024 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:13.490446 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:16:13.490552 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:16:13.490646 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:13.490768 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 20:16:13.490850 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 20:16:13.494168 13340 raft_consensus.cc:1261] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Refusing update from remote peer 3efea12b7cbb41ada3f316cab53be5b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:13.494189 13485 raft_consensus.cc:1261] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Refusing update from remote peer 3efea12b7cbb41ada3f316cab53be5b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:13.494800 13584 consensus_queue.cc:1035] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:13.500916 13584 consensus_queue.cc:1035] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:13.540894 13043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:13.541136 13043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:13.544150 13043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb: Sending DeleteTablet for 3 replicas of tablet b96b30d4f3b1411f90aaeb016792fb66
I20260923 20:16:13.544931 13182 tablet_service.cc:1514] Processing DeleteTablet for tablet b96b30d4f3b1411f90aaeb016792fb66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:13 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:52236
I20260923 20:16:13.545032 13320 tablet_service.cc:1514] Processing DeleteTablet for tablet b96b30d4f3b1411f90aaeb016792fb66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:13 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:40218
I20260923 20:16:13.545231 13623 tablet_replica.cc:330] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9: stopping tablet replica
I20260923 20:16:13.545334 13621 tablet_replica.cc:330] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338: stopping tablet replica
I20260923 20:16:13.545446 13623 raft_consensus.cc:2229] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:16:13.545528 13621 raft_consensus.cc:2229] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:13.545753 13621 raft_consensus.cc:2258] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:13.545768 13623 raft_consensus.cc:2258] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:13.546586 13460 tablet_service.cc:1514] Processing DeleteTablet for tablet b96b30d4f3b1411f90aaeb016792fb66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:13 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:52950
I20260923 20:16:13.546923 13621 ts_tablet_manager.cc:1905] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:13.547123 13043 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 20:16:13.547340 13623 ts_tablet_manager.cc:1905] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 20:16:13.547868 13043 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:16:13.549474 13622 tablet_replica.cc:330] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479: stopping tablet replica
I20260923 20:16:13.549713 13622 raft_consensus.cc:2229] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:13.549966 13622 raft_consensus.cc:2258] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:13.551061 13622 ts_tablet_manager.cc:1905] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:13.554697 13623 ts_tablet_manager.cc:1918] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:13.554921 13623 log.cc:1196] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/b96b30d4f3b1411f90aaeb016792fb66
I20260923 20:16:13.555441 13623 ts_tablet_manager.cc:1939] T b96b30d4f3b1411f90aaeb016792fb66 P 3efea12b7cbb41ada3f316cab53be5b9: Deleting consensus metadata
I20260923 20:16:13.556779 13031 catalog_manager.cc:4841] TS 3efea12b7cbb41ada3f316cab53be5b9 (127.5.83.65:36757): tablet b96b30d4f3b1411f90aaeb016792fb66 (table test-table [id=6437c4c9d21247e883b16ed74cfc2522]) successfully deleted
I20260923 20:16:13.557679 13622 ts_tablet_manager.cc:1918] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:13.557839 13622 log.cc:1196] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/b96b30d4f3b1411f90aaeb016792fb66
I20260923 20:16:13.558254 13622 ts_tablet_manager.cc:1939] T b96b30d4f3b1411f90aaeb016792fb66 P 4b87c71baf544daabf85c48235a0a479: Deleting consensus metadata
I20260923 20:16:13.558506 13621 ts_tablet_manager.cc:1918] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:13.558662 13621 log.cc:1196] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/b96b30d4f3b1411f90aaeb016792fb66
I20260923 20:16:13.559064 13621 ts_tablet_manager.cc:1939] T b96b30d4f3b1411f90aaeb016792fb66 P 50ea93ba0f394262814ef7f6202be338: Deleting consensus metadata
I20260923 20:16:13.560304 13032 catalog_manager.cc:4841] TS 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643): tablet b96b30d4f3b1411f90aaeb016792fb66 (table test-table [id=6437c4c9d21247e883b16ed74cfc2522]) successfully deleted
I20260923 20:16:13.561587 13032 catalog_manager.cc:4841] TS 4b87c71baf544daabf85c48235a0a479 (127.5.83.67:42127): tablet b96b30d4f3b1411f90aaeb016792fb66 (table test-table [id=6437c4c9d21247e883b16ed74cfc2522]) successfully deleted
I20260923 20:16:13.561720 13182 tablet_service.cc:1467] Processing CreateTablet for tablet cd252bf5a8db4c34b3eaa8df7564a4e8 (DEFAULT_TABLE table=test-table [id=64d76449cb0b4923a0e60d815fd72f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:13.562012 13320 tablet_service.cc:1467] Processing CreateTablet for tablet cd252bf5a8db4c34b3eaa8df7564a4e8 (DEFAULT_TABLE table=test-table [id=64d76449cb0b4923a0e60d815fd72f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:13.562196 13182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd252bf5a8db4c34b3eaa8df7564a4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:13.562390 13320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd252bf5a8db4c34b3eaa8df7564a4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:13.567013 13637 tablet_bootstrap.cc:492] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9: Bootstrap starting.
I20260923 20:16:13.567368 13460 tablet_service.cc:1467] Processing CreateTablet for tablet cd252bf5a8db4c34b3eaa8df7564a4e8 (DEFAULT_TABLE table=test-table [id=64d76449cb0b4923a0e60d815fd72f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:13.567783 13460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd252bf5a8db4c34b3eaa8df7564a4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:13.568567 13638 tablet_bootstrap.cc:492] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338: Bootstrap starting.
I20260923 20:16:13.568703 13637 tablet_bootstrap.cc:654] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:13.569957 13638 tablet_bootstrap.cc:654] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:13.570923 13637 tablet_bootstrap.cc:492] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9: No bootstrap required, opened a new log
I20260923 20:16:13.571126 13637 ts_tablet_manager.cc:1397] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260923 20:16:13.571925 13637 raft_consensus.cc:348] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.572221 13637 raft_consensus.cc:374] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:13.572319 13637 raft_consensus.cc:729] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efea12b7cbb41ada3f316cab53be5b9, State: Initialized, Role: FOLLOWER
I20260923 20:16:13.572610 13637 consensus_queue.cc:260] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.573374 13637 ts_tablet_manager.cc:1428] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 20:16:13.573549 13638 tablet_bootstrap.cc:492] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338: No bootstrap required, opened a new log
I20260923 20:16:13.573716 13638 ts_tablet_manager.cc:1397] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260923 20:16:13.574389 13638 raft_consensus.cc:348] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.574604 13638 raft_consensus.cc:374] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:13.574687 13638 raft_consensus.cc:729] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ea93ba0f394262814ef7f6202be338, State: Initialized, Role: FOLLOWER
I20260923 20:16:13.574921 13638 consensus_queue.cc:260] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.575454 13638 ts_tablet_manager.cc:1428] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 20:16:13.575515 13639 tablet_bootstrap.cc:492] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479: Bootstrap starting.
I20260923 20:16:13.577215 13639 tablet_bootstrap.cc:654] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:13.579696 13639 tablet_bootstrap.cc:492] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479: No bootstrap required, opened a new log
I20260923 20:16:13.579862 13639 ts_tablet_manager.cc:1397] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 20:16:13.580610 13639 raft_consensus.cc:348] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.580893 13639 raft_consensus.cc:374] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:13.580994 13639 raft_consensus.cc:729] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b87c71baf544daabf85c48235a0a479, State: Initialized, Role: FOLLOWER
I20260923 20:16:13.581245 13639 consensus_queue.cc:260] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.581729 13639 ts_tablet_manager.cc:1428] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 20:16:13.954373 13584 raft_consensus.cc:482] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:16:13.954566 13584 raft_consensus.cc:504] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.955144 13584 leader_election.cc:288] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643), 4b87c71baf544daabf85c48235a0a479 (127.5.83.67:42127)
I20260923 20:16:13.955734 13340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd252bf5a8db4c34b3eaa8df7564a4e8" candidate_uuid: "3efea12b7cbb41ada3f316cab53be5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ea93ba0f394262814ef7f6202be338" is_pre_election: true
I20260923 20:16:13.955870 13485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd252bf5a8db4c34b3eaa8df7564a4e8" candidate_uuid: "3efea12b7cbb41ada3f316cab53be5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b87c71baf544daabf85c48235a0a479" is_pre_election: true
I20260923 20:16:13.955978 13340 raft_consensus.cc:2454] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efea12b7cbb41ada3f316cab53be5b9 in term 0.
I20260923 20:16:13.956115 13485 raft_consensus.cc:2454] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efea12b7cbb41ada3f316cab53be5b9 in term 0.
I20260923 20:16:13.956470 13135 leader_election.cc:302] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efea12b7cbb41ada3f316cab53be5b9, 50ea93ba0f394262814ef7f6202be338; no voters:
I20260923 20:16:13.956840 13584 raft_consensus.cc:2781] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:16:13.956950 13584 raft_consensus.cc:482] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:16:13.957093 13584 raft_consensus.cc:3037] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:13.959230 13584 raft_consensus.cc:504] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.959825 13584 leader_election.cc:288] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [CANDIDATE]: Term 1 election: Requested vote from peers 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643), 4b87c71baf544daabf85c48235a0a479 (127.5.83.67:42127)
I20260923 20:16:13.960219 13340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd252bf5a8db4c34b3eaa8df7564a4e8" candidate_uuid: "3efea12b7cbb41ada3f316cab53be5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ea93ba0f394262814ef7f6202be338"
I20260923 20:16:13.960347 13485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd252bf5a8db4c34b3eaa8df7564a4e8" candidate_uuid: "3efea12b7cbb41ada3f316cab53be5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b87c71baf544daabf85c48235a0a479"
I20260923 20:16:13.960426 13340 raft_consensus.cc:3037] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:13.960577 13485 raft_consensus.cc:3037] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:13.962612 13340 raft_consensus.cc:2454] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efea12b7cbb41ada3f316cab53be5b9 in term 1.
I20260923 20:16:13.962967 13135 leader_election.cc:302] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efea12b7cbb41ada3f316cab53be5b9, 50ea93ba0f394262814ef7f6202be338; no voters:
I20260923 20:16:13.963280 13584 raft_consensus.cc:2781] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:13.963464 13584 raft_consensus.cc:686] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 LEADER]: Becoming Leader. State: Replica: 3efea12b7cbb41ada3f316cab53be5b9, State: Running, Role: LEADER
I20260923 20:16:13.963479 13485 raft_consensus.cc:2454] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efea12b7cbb41ada3f316cab53be5b9 in term 1.
I20260923 20:16:13.963778 13584 consensus_queue.cc:237] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } }
I20260923 20:16:13.966507 13043 catalog_manager.cc:5495] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efea12b7cbb41ada3f316cab53be5b9 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "3efea12b7cbb41ada3f316cab53be5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efea12b7cbb41ada3f316cab53be5b9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 36757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 } health_report { overall_health: UNKNOWN } } }
I20260923 20:16:14.050756 13485 raft_consensus.cc:1261] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Refusing update from remote peer 3efea12b7cbb41ada3f316cab53be5b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:14.050915 13340 raft_consensus.cc:1261] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Refusing update from remote peer 3efea12b7cbb41ada3f316cab53be5b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:14.051496 13562 consensus_queue.cc:1035] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ea93ba0f394262814ef7f6202be338" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:14.051811 13584 consensus_queue.cc:1035] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b87c71baf544daabf85c48235a0a479" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:14.097415 13044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:14.097668 13044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:14.100579 13044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a3618d015a240e3a4bb3d4422ad11bb: Sending DeleteTablet for 3 replicas of tablet cd252bf5a8db4c34b3eaa8df7564a4e8
I20260923 20:16:14.101393 13182 tablet_service.cc:1514] Processing DeleteTablet for tablet cd252bf5a8db4c34b3eaa8df7564a4e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:14 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:52236
I20260923 20:16:14.101462 13320 tablet_service.cc:1514] Processing DeleteTablet for tablet cd252bf5a8db4c34b3eaa8df7564a4e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:14 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:40218
I20260923 20:16:14.101552 13460 tablet_service.cc:1514] Processing DeleteTablet for tablet cd252bf5a8db4c34b3eaa8df7564a4e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:14 UTC) from {username='oryx', principal='oryx/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:52950
I20260923 20:16:14.102174 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 13116
I20260923 20:16:14.113211 13663 tablet_replica.cc:330] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9: stopping tablet replica
I20260923 20:16:14.113672 13663 raft_consensus.cc:2229] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:16:14.114274 13663 raft_consensus.cc:2258] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:14.115633 13663 ts_tablet_manager.cc:1905] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:14.116323 13664 tablet_replica.cc:330] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338: stopping tablet replica
I20260923 20:16:14.116596 13664 raft_consensus.cc:2229] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:14.116900 13664 raft_consensus.cc:2258] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:14.118019 13664 ts_tablet_manager.cc:1905] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:14.120945 13665 tablet_replica.cc:330] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479: stopping tablet replica
I20260923 20:16:14.121189 13665 raft_consensus.cc:2229] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:14.121445 13665 raft_consensus.cc:2258] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:14.122542 13665 ts_tablet_manager.cc:1905] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 20:16:14 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.65@KRBTEST.COM
I20260923 20:16:14.124181 13663 ts_tablet_manager.cc:1918] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:16:14.124380 13663 log.cc:1196] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/cd252bf5a8db4c34b3eaa8df7564a4e8
I20260923 20:16:14.124928 13664 ts_tablet_manager.cc:1918] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:16:14.124989 13663 ts_tablet_manager.cc:1939] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 3efea12b7cbb41ada3f316cab53be5b9: Deleting consensus metadata
I20260923 20:16:14.125083 13664 log.cc:1196] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/cd252bf5a8db4c34b3eaa8df7564a4e8
I20260923 20:16:14.125554 13664 ts_tablet_manager.cc:1939] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 50ea93ba0f394262814ef7f6202be338: Deleting consensus metadata
I20260923 20:16:14.126343 13031 catalog_manager.cc:4841] TS 3efea12b7cbb41ada3f316cab53be5b9 (127.5.83.65:36757): tablet cd252bf5a8db4c34b3eaa8df7564a4e8 (table test-table [id=64d76449cb0b4923a0e60d815fd72f03]) successfully deleted
I20260923 20:16:14.126942 13032 catalog_manager.cc:4841] TS 50ea93ba0f394262814ef7f6202be338 (127.5.83.66:40643): tablet cd252bf5a8db4c34b3eaa8df7564a4e8 (table test-table [id=64d76449cb0b4923a0e60d815fd72f03]) successfully deleted
I20260923 20:16:14.128715 13665 ts_tablet_manager.cc:1918] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 20:16:14.128860 13665 log.cc:1196] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/cd252bf5a8db4c34b3eaa8df7564a4e8
I20260923 20:16:14.129232 13665 ts_tablet_manager.cc:1939] T cd252bf5a8db4c34b3eaa8df7564a4e8 P 4b87c71baf544daabf85c48235a0a479: Deleting consensus metadata
I20260923 20:16:14.130060 13032 catalog_manager.cc:4841] TS 4b87c71baf544daabf85c48235a0a479 (127.5.83.67:42127): tablet cd252bf5a8db4c34b3eaa8df7564a4e8 (table test-table [id=64d76449cb0b4923a0e60d815fd72f03]) successfully deleted
I20260923 20:16:14.131753 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:14.103767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.5.83.65:36757)
0923 20:16:14.113275 (+ 9508us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:14.113282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:14.113559 (+ 277us) server_negotiation.cc:407] Connection header received
0923 20:16:14.113788 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:14.113794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:14.113880 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:14.113998 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:14.115993 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:14.117213 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:14.119027 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:14.119677 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:14.119797 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:14.125271 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:14.125294 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:14.125317 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:14.125356 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:14.128843 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:14.129618 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:14.129633 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:14.129635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:14.129761 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:14.130189 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:14.130197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:14.130200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:14.130583 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:14.130867 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:14.131174 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 20:16:14.131392 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8893,"thread_start_us":8756,"threads_started":1}
I20260923 20:16:14.132351 13246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:14.257998 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 13116
I20260923 20:16:14.282760 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 13256
Sep 23 20:16:14 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.66@KRBTEST.COM
I20260923 20:16:14.297683 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:14.283579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.5.83.66:40643)
0923 20:16:14.284113 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:14.284117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:14.284136 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:16:14.284198 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:14.284202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:14.284273 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:14.284382 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:14.285626 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:14.286472 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:14.287672 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:14.288100 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:14.288202 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:14.292326 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:14.292339 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:14.292354 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:14.292386 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:14.295154 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:14.295756 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:14.295763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:14.295765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:14.295878 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:14.296222 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:14.296227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:14.296229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:14.296751 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:14.297005 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:14.297229 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 20:16:14.297407 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":296,"threads_started":1}
I20260923 20:16:14.298468 13386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:14.396442 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 13256
I20260923 20:16:14.414733 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 13395
Sep 23 20:16:14 dist-test-slave-vl24 krb5kdc[13000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.83.67@KRBTEST.COM
I20260923 20:16:14.429816 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:14.415490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.5.83.67:42127)
0923 20:16:14.416063 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:14.416076 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:14.416092 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:16:14.416147 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:14.416151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:14.416218 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:14.416314 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:14.417600 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:14.418427 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:14.419607 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:14.420003 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:14.420098 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:14.424316 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:14.424332 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:14.424348 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:14.424383 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:14.427342 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:14.427975 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:14.427982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:14.427984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:14.428111 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:14.428405 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:14.428408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:14.428410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:14.428799 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:14.429022 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:14.429220 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 20:16:14.429404 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":345,"threads_started":1}
I20260923 20:16:14.430464 13525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:14.576648 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 13395
I20260923 20:16:14.602005 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 13013
I20260923 20:16:14.603166 13085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:14.720954 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 13013
2026-09-23T20:16:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6174 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:16:14 dist-test-slave-vl24 krb5kdc[13676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:16:14 dist-test-slave-vl24 krb5kdc[13676](info): set up 2 sockets
Sep 23 20:16:14 dist-test-slave-vl24 krb5kdc[13676](info): commencing operation
krb5kdc: starting...
W20260923 20:16:15.826920 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:16:15 dist-test-slave-vl24 krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:16:16.060467 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:37061
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:42537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:37061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:16.400532 13689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:16.400911 13689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:16.401055 13689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:16.408561 13689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:16:16.408718 13689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:16.408815 13689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:16.408931 13689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:16:16.409024 13689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:16:16.419402 13689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:42537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:37061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:37061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:16.420840 13689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:16.422326 13689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:16.435108 13697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:16.435148 13695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:16.435070 13694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:16.435899 13689 server_base.cc:1034] running on GCE node
I20260923 20:16:16.436779 13689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:16.438248 13689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:16.439458 13689 hybrid_clock.cc:648] HybridClock initialized: now 1790194576439385 us; error 55 us; skew 500 ppm
I20260923 20:16:16.439813 13689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T20:16:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1787 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:16:16 dist-test-slave-vl24 krb5kdc[13703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:16:16 dist-test-slave-vl24 krb5kdc[13703](info): set up 2 sockets
Sep 23 20:16:16 dist-test-slave-vl24 krb5kdc[13703](info): commencing operation
krb5kdc: starting...
W20260923 20:16:17.638980 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:16:17 dist-test-slave-vl24 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:16:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:16:17.864502 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:43205
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:43205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:18.198800 13716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:18.199133 13716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:18.199306 13716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:18.206862 13716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:16:18.206985 13716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:18.207118 13716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:18.207211 13716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:16:18.207298 13716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:16:18.217270 13716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:43205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:43205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:18.218649 13716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:18.219934 13716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:18.231621 13724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:18.231667 13722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:18.231711 13721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:18.232517 13716 server_base.cc:1034] running on GCE node
I20260923 20:16:18.233300 13716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:18.234563 13716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:18.235764 13716 hybrid_clock.cc:648] HybridClock initialized: now 1790194578235702 us; error 50 us; skew 500 ppm
I20260923 20:16:18.236102 13716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:18 dist-test-slave-vl24 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194578, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:18.239094 13716 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:16:18.240998 13716 webserver.cc:460] Webserver started at http://127.5.83.126:39801/ using document root <none> and password file <none>
I20260923 20:16:18.241696 13716 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:18.241834 13716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:18.242218 13716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:18.245460 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "1d20c83d429f4a278d1246ae5a71dfb3"
format_stamp: "Formatted at 2026-09-23 20:16:18 on dist-test-slave-vl24"
server_key: "6b6cd93c6c1cb356a0625c16acaec4db"
server_key_iv: "ee072485dbb4cda2e054b19dcc9e469f"
server_key_version: "encryptionkey@0"
I20260923 20:16:18.246318 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "1d20c83d429f4a278d1246ae5a71dfb3"
format_stamp: "Formatted at 2026-09-23 20:16:18 on dist-test-slave-vl24"
server_key: "6b6cd93c6c1cb356a0625c16acaec4db"
server_key_iv: "ee072485dbb4cda2e054b19dcc9e469f"
server_key_version: "encryptionkey@0"
I20260923 20:16:18.251848 13716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:16:18.255741 13731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:18.257189 13716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:16:18.257375 13716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "1d20c83d429f4a278d1246ae5a71dfb3"
format_stamp: "Formatted at 2026-09-23 20:16:18 on dist-test-slave-vl24"
server_key: "6b6cd93c6c1cb356a0625c16acaec4db"
server_key_iv: "ee072485dbb4cda2e054b19dcc9e469f"
server_key_version: "encryptionkey@0"
I20260923 20:16:18.257576 13716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:18.276398 13716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:18.280236 13716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:18.280525 13716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:18.298866 13716 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:43205
I20260923 20:16:18.298902 13782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:43205 every 8 connection(s)
I20260923 20:16:18.300225 13716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:16:18.304070 13783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:18.306461 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 13716
I20260923 20:16:18.306704 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:16:18.307502 5453 external_mini_cluster.cc:1428] Setting key 4146f3164636997c8a48763c8684eef1
I20260923 20:16:18.316656 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3: Bootstrap starting.
I20260923 20:16:18.320331 13783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:18.321686 13783 log.cc:824] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:16:18 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:16:18.324815 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3: No bootstrap required, opened a new log
I20260923 20:16:18.332739 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:18.309676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.5.83.126:43205)
0923 20:16:18.310567 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:18.310578 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:18.310635 (+ 57us) server_negotiation.cc:407] Connection header received
0923 20:16:18.312418 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:18.312442 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:18.312925 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:18.313265 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:18.315065 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:18.316444 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:18.318226 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:18.318919 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:18.319051 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:18.324003 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:18.324014 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:18.324031 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:18.324071 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:18.328156 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:18.328888 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:18.328897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:18.328906 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:18.329127 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:18.329464 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:18.329469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:18.329473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:18.330188 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:18.330491 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:18.330869 (+ 378us) server_negotiation.cc:299] Negotiation successful
0923 20:16:18.331316 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":444,"threads_started":1}
I20260923 20:16:18.335367 13783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43205 } }
I20260923 20:16:18.335764 13783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:18.335894 13783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d20c83d429f4a278d1246ae5a71dfb3, State: Initialized, Role: FOLLOWER
I20260923 20:16:18.336472 13783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43205 } }
I20260923 20:16:18.336768 13783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:16:18.336910 13783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:16:18.337075 13783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:18.339167 13783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43205 } }
I20260923 20:16:18.339655 13783 leader_election.cc:302] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d20c83d429f4a278d1246ae5a71dfb3; no voters:
I20260923 20:16:18.340248 13783 leader_election.cc:288] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:16:18.340390 13788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:18.341540 13788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 1 LEADER]: Becoming Leader. State: Replica: 1d20c83d429f4a278d1246ae5a71dfb3, State: Running, Role: LEADER
I20260923 20:16:18.342141 13788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43205 } }
I20260923 20:16:18.342588 13783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:16:18.345173 13789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43205 } } }
I20260923 20:16:18.345542 13789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:18.345279 13790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d20c83d429f4a278d1246ae5a71dfb3. Latest consensus state: current_term: 1 leader_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43205 } } }
I20260923 20:16:18.345669 13790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:18.346473 13795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:16:18.353564 13795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:16:18.363307 13795 catalog_manager.cc:1347] Generated new cluster ID: 24ad9a3cfc164f83854f1f4e56ee1093
I20260923 20:16:18.363426 13795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:16:18.381762 13795 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:16:18.383076 13795 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:16:18.395581 13795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3: Generated new TSK 0
I20260923 20:16:18.396689 13795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:16:18.453603 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:43205
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:18.784925 13809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:18.785221 13809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:18.785377 13809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:18.792708 13809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:18.792824 13809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:18.793023 13809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:16:18.802963 13809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:43205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:18.804292 13809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:18.805670 13809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:18.818285 13814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:18.818406 13815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:18.818521 13817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:18.819762 13809 server_base.cc:1034] running on GCE node
I20260923 20:16:18.820307 13809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:18.821231 13809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:18.822427 13809 hybrid_clock.cc:648] HybridClock initialized: now 1790194578822384 us; error 35 us; skew 500 ppm
I20260923 20:16:18.822739 13809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:18 dist-test-slave-vl24 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194578, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:18.825845 13809 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:16:18.827402 13809 webserver.cc:460] Webserver started at http://127.5.83.65:32983/ using document root <none> and password file <none>
I20260923 20:16:18.828071 13809 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:18.828199 13809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:18.828500 13809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:18.831382 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "769f91f86de24654a6eccff2c6002b30"
format_stamp: "Formatted at 2026-09-23 20:16:18 on dist-test-slave-vl24"
server_key: "ae6f33d1767acf4f959dbed974e460e1"
server_key_iv: "29dc6be491634d53f14648518b3ab53d"
server_key_version: "encryptionkey@0"
I20260923 20:16:18.832149 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "769f91f86de24654a6eccff2c6002b30"
format_stamp: "Formatted at 2026-09-23 20:16:18 on dist-test-slave-vl24"
server_key: "ae6f33d1767acf4f959dbed974e460e1"
server_key_iv: "29dc6be491634d53f14648518b3ab53d"
server_key_version: "encryptionkey@0"
I20260923 20:16:18.837371 13809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 20:16:18.840829 13824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:18.842242 13809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:16:18.842396 13809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "769f91f86de24654a6eccff2c6002b30"
format_stamp: "Formatted at 2026-09-23 20:16:18 on dist-test-slave-vl24"
server_key: "ae6f33d1767acf4f959dbed974e460e1"
server_key_iv: "29dc6be491634d53f14648518b3ab53d"
server_key_version: "encryptionkey@0"
I20260923 20:16:18.842612 13809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:18.874585 13809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:18.878962 13809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:18.879254 13809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:18.880234 13809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:18.881621 13809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:18.881696 13809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:18.881861 13809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:18.881924 13809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:18.914628 13809 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:32997
I20260923 20:16:18.914671 13936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:32997 every 8 connection(s)
I20260923 20:16:18.915934 13809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:16:18.917333 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 13809
I20260923 20:16:18.917547 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:16:18.918350 5453 external_mini_cluster.cc:1428] Setting key 844519fb5c50e565bfb794f35ece4acb
Sep 23 20:16:18 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194578, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:16:18.936769 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:18.918917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:52689 (local address 127.5.83.126:43205)
0923 20:16:18.919385 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:18.919391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:18.920406 (+ 1015us) server_negotiation.cc:407] Connection header received
0923 20:16:18.922297 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:18.922305 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:18.922397 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:18.922512 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:18.923924 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:18.925075 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:18.926613 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:18.927338 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:18.927477 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:18.931167 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:18.931172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:18.931175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:18.931207 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:18.933684 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:18.934413 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:18.934420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:18.934423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:18.934555 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:18.934984 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:18.934992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:18.934995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:18.935262 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:18.935418 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:18.936129 (+ 711us) server_negotiation.cc:299] Negotiation successful
0923 20:16:18.936358 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":245,"threads_started":1}
I20260923 20:16:18.937776 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:18.919460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:52689)
0923 20:16:18.920261 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:18.920308 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:18.922029 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:18.922711 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:18.922719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:18.923203 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:18.923758 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:18.923776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:18.925231 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:18.925234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:18.926475 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:18.926487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:18.927574 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:18.927578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:18.927781 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:18.928485 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:18.928506 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:18.930917 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:18.933833 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:18.933842 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:18.933860 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:18.934293 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:18.934666 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:18.934671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:18.934675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:18.934869 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:18.935386 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:18.935394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:18.935736 (+ 342us) client_negotiation.cc:769] Sending connection context
0923 20:16:18.936005 (+ 269us) client_negotiation.cc:240] Negotiation successful
0923 20:16:18.936579 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":344,"threads_started":1}
I20260923 20:16:18.939430 13937 heartbeater.cc:347] Connected to a master server at 127.5.83.126:43205
I20260923 20:16:18.939785 13937 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:18.940753 13937 heartbeater.cc:511] Master 127.5.83.126:43205 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
I20260923 20:16:18.943105 13748 ts_manager.cc:194] Registered new tserver with Master: 769f91f86de24654a6eccff2c6002b30 (127.5.83.65:32997)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
I20260923 20:16:18.944867 13748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:52689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:16:18.972632 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:43205
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:19.313081 13943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:19.313340 13943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:19.313436 13943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:19.320565 13943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:19.320676 13943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:19.320786 13943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:16:19.330331 13943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:43205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:19.331624 13943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:19.333123 13943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:19.346401 13948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:19.346351 13951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:19.346416 13949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:19.347512 13943 server_base.cc:1034] running on GCE node
I20260923 20:16:19.348141 13943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:19.349192 13943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:19.350447 13943 hybrid_clock.cc:648] HybridClock initialized: now 1790194579350395 us; error 33 us; skew 500 ppm
I20260923 20:16:19.350817 13943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:19 dist-test-slave-vl24 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194579, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:19.354231 13943 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:16:19.355747 13943 webserver.cc:460] Webserver started at http://127.5.83.66:41695/ using document root <none> and password file <none>
I20260923 20:16:19.356485 13943 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:19.356659 13943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:19.357017 13943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:19.360246 13943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "650bc7c075fc4d5bab67854cd27dfd89"
format_stamp: "Formatted at 2026-09-23 20:16:19 on dist-test-slave-vl24"
server_key: "9456e3dbb7749545f2ec5aec74b75418"
server_key_iv: "e21d6af4c23a433b89bba6b19d8d4d1b"
server_key_version: "encryptionkey@0"
I20260923 20:16:19.361160 13943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "650bc7c075fc4d5bab67854cd27dfd89"
format_stamp: "Formatted at 2026-09-23 20:16:19 on dist-test-slave-vl24"
server_key: "9456e3dbb7749545f2ec5aec74b75418"
server_key_iv: "e21d6af4c23a433b89bba6b19d8d4d1b"
server_key_version: "encryptionkey@0"
I20260923 20:16:19.366768 13943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:16:19.370399 13958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:19.371819 13943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:16:19.371996 13943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "650bc7c075fc4d5bab67854cd27dfd89"
format_stamp: "Formatted at 2026-09-23 20:16:19 on dist-test-slave-vl24"
server_key: "9456e3dbb7749545f2ec5aec74b75418"
server_key_iv: "e21d6af4c23a433b89bba6b19d8d4d1b"
server_key_version: "encryptionkey@0"
I20260923 20:16:19.372210 13943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:19.387091 13943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:19.391815 13943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:19.392154 13943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:19.393292 13943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:19.394783 13943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:19.394863 13943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:19.395048 13943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:19.395135 13943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:19.431497 13943 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:41605
I20260923 20:16:19.431584 14070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:41605 every 8 connection(s)
I20260923 20:16:19.432888 13943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:16:19.436206 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 13943
I20260923 20:16:19.436450 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:16:19.437103 5453 external_mini_cluster.cc:1428] Setting key be7cc9f19d5ebf6fd8c670c65e9d7e32
Sep 23 20:16:19 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194579, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:16:19.455170 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:19.435584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:55985 (local address 127.5.83.126:43205)
0923 20:16:19.435754 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:19.435758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:19.437207 (+ 1449us) server_negotiation.cc:407] Connection header received
0923 20:16:19.439418 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:19.439423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:19.439521 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:19.439624 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:19.441166 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:19.441895 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:19.443526 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:19.444043 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:19.444174 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:19.447828 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:19.447833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:19.447837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:19.447870 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:19.451536 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:19.452342 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:19.452349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:19.452356 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:19.452576 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:19.453066 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:19.453074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:19.453080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:19.453465 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:19.453642 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:19.454420 (+ 778us) server_negotiation.cc:299] Negotiation successful
0923 20:16:19.454708 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 20:16:19.455852 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:19.436162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:55985)
0923 20:16:19.437033 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:19.437078 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:19.439161 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:19.439852 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:19.439861 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:19.440365 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:19.440999 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:19.441025 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:19.442059 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:19.442063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:19.443371 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:19.443382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:19.444277 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:19.444284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:19.444562 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:19.445287 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:19.445309 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:19.447603 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:19.451707 (+ 4104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:19.451715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:19.451729 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:19.452205 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:19.452706 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:19.452710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:19.452712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:19.452943 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:19.453633 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:19.453640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:19.454029 (+ 389us) client_negotiation.cc:769] Sending connection context
0923 20:16:19.454276 (+ 247us) client_negotiation.cc:240] Negotiation successful
0923 20:16:19.454689 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":410,"threads_started":1}
I20260923 20:16:19.457477 14071 heartbeater.cc:347] Connected to a master server at 127.5.83.126:43205
I20260923 20:16:19.457836 14071 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:19.458743 14071 heartbeater.cc:511] Master 127.5.83.126:43205 requested a full tablet report, sending...
I20260923 20:16:19.460417 13748 ts_manager.cc:194] Registered new tserver with Master: 650bc7c075fc4d5bab67854cd27dfd89 (127.5.83.66:41605)
I20260923 20:16:19.461331 13748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:55985
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:16:19.493800 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:43205
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:19.823078 14076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:19.823369 14076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:19.823464 14076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:19.830806 14076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:19.830905 14076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:19.831017 14076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:16:19.841015 14076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:43205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:19.842303 14076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:19.843641 14076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:19.857127 14081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:19.857089 14084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:19.857052 14082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:19.858180 14076 server_base.cc:1034] running on GCE node
I20260923 20:16:19.858786 14076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:19.859719 14076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:19.861131 14076 hybrid_clock.cc:648] HybridClock initialized: now 1790194579861078 us; error 34 us; skew 500 ppm
I20260923 20:16:19.861464 14076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:19 dist-test-slave-vl24 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194579, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:19.864727 14076 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:16:19.866335 14076 webserver.cc:460] Webserver started at http://127.5.83.67:40641/ using document root <none> and password file <none>
I20260923 20:16:19.867157 14076 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:19.867312 14076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:19.867656 14076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:19.870903 14076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "af4f91afcfe74a0bbd056b498b9a892c"
format_stamp: "Formatted at 2026-09-23 20:16:19 on dist-test-slave-vl24"
server_key: "d056f361b4ad06d8990712ad6579c88a"
server_key_iv: "358963b58e1b6d85b8e22bc9d7644dec"
server_key_version: "encryptionkey@0"
I20260923 20:16:19.871763 14076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "af4f91afcfe74a0bbd056b498b9a892c"
format_stamp: "Formatted at 2026-09-23 20:16:19 on dist-test-slave-vl24"
server_key: "d056f361b4ad06d8990712ad6579c88a"
server_key_iv: "358963b58e1b6d85b8e22bc9d7644dec"
server_key_version: "encryptionkey@0"
I20260923 20:16:19.877956 14076 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 20:16:19.881850 14091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:19.883296 14076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:16:19.883500 14076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "af4f91afcfe74a0bbd056b498b9a892c"
format_stamp: "Formatted at 2026-09-23 20:16:19 on dist-test-slave-vl24"
server_key: "d056f361b4ad06d8990712ad6579c88a"
server_key_iv: "358963b58e1b6d85b8e22bc9d7644dec"
server_key_version: "encryptionkey@0"
I20260923 20:16:19.883716 14076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:19.911190 14076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:19.916137 14076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:19.916466 14076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:19.917739 14076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:19.919539 14076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:19.919633 14076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:19.919752 14076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:19.919816 14076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:19.947962 13937 heartbeater.cc:503] Master 127.5.83.126:43205 was elected leader, sending a full tablet report...
I20260923 20:16:19.952467 14076 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:36453
I20260923 20:16:19.952504 14203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:36453 every 8 connection(s)
I20260923 20:16:19.953714 14076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:16:19.958319 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 14076
I20260923 20:16:19.958505 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:16:19.958928 5453 external_mini_cluster.cc:1428] Setting key fa7cd94b9e872cf2b32d38874f53e2a0
Sep 23 20:16:19 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194579, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:16:19.973963 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:19.956097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:45305 (local address 127.5.83.126:43205)
0923 20:16:19.956613 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:19.956620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:19.957527 (+ 907us) server_negotiation.cc:407] Connection header received
0923 20:16:19.959529 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:19.959536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:19.959625 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:19.959714 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:19.961481 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:19.962355 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:19.963822 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:19.964215 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:19.964343 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:19.968031 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:19.968036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:19.968040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:19.968075 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:19.970701 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:19.971467 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:19.971477 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:19.971481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:19.971601 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:19.972048 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:19.972053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:19.972057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:19.972340 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:19.972507 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:19.973240 (+ 733us) server_negotiation.cc:299] Negotiation successful
0923 20:16:19.973477 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":207,"threads_started":1}
I20260923 20:16:19.974965 14207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:19.956718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:45305)
0923 20:16:19.957380 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:19.957423 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:19.959248 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:19.959945 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:19.959953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:19.960600 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:19.961248 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:19.961265 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:19.962494 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:19.962504 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:19.963681 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:19.963696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:19.964434 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:19.964443 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:19.964718 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:19.965429 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:19.965457 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:19.967799 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:19.970855 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:19.970864 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:19.970882 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:19.971326 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:19.971719 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:19.971725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:19.971729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:19.971926 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:19.972480 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:19.972489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:19.972853 (+ 364us) client_negotiation.cc:769] Sending connection context
0923 20:16:19.973128 (+ 275us) client_negotiation.cc:240] Negotiation successful
0923 20:16:19.973707 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":263,"threads_started":1}
I20260923 20:16:19.976423 14204 heartbeater.cc:347] Connected to a master server at 127.5.83.126:43205
I20260923 20:16:19.976759 14204 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:19.977560 14204 heartbeater.cc:511] Master 127.5.83.126:43205 requested a full tablet report, sending...
I20260923 20:16:19.978925 13748 ts_manager.cc:194] Registered new tserver with Master: af4f91afcfe74a0bbd056b498b9a892c (127.5.83.67:36453)
I20260923 20:16:19.979722 13748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:45305
I20260923 20:16:19.986219 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab.
Entry for principal oryx/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab.
WARNING: no policy specified for oryx/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab.
Entry for principal oryx/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab.
WARNING: no policy specified for oryx/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab.
Entry for principal oryx/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab.
I20260923 20:16:20.129212 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 13809
Sep 23 20:16:20 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:16:20.145941 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:20.130470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.5.83.65:32997)
0923 20:16:20.131144 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:20.131152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:20.131180 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:16:20.131243 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:20.131250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:20.131474 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:20.131610 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:20.133129 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:20.134280 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:20.135563 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:20.136158 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:20.136270 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:20.140316 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:20.140329 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:20.140343 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:20.140376 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:20.143347 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:20.144023 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:20.144030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:20.144032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:20.144165 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:20.144470 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:20.144473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:20.144475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:20.144850 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:20.145091 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:20.145294 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 20:16:20.145545 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":336,"threads_started":1}
I20260923 20:16:20.147253 13932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:20.232343 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 13809
I20260923 20:16:20.252096 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 13943
Sep 23 20:16:20 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:16:20.268255 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:20.252997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.5.83.66:41605)
0923 20:16:20.253571 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:20.253578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:20.253603 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:16:20.253663 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:20.253670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:20.253882 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:20.254007 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:20.255188 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:20.256175 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:20.257619 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:20.258172 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:20.258271 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:20.262700 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:20.262712 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:20.262727 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:20.262764 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:20.265648 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:20.266333 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:20.266339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:20.266341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:20.266457 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:20.266882 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:20.266886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:20.266888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:20.267221 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:20.267449 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:20.267760 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 20:16:20.267964 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":295,"threads_started":1}
I20260923 20:16:20.269726 14066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:20.360275 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 13943
I20260923 20:16:20.376438 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 14076
Sep 23 20:16:20 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:16:20.395041 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:20.377422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52428 (local address 127.5.83.67:36453)
0923 20:16:20.378021 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:20.378028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:20.378057 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:16:20.378117 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:20.378124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:20.378350 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:20.378527 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:20.379834 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:20.381083 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:20.382311 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:20.382860 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:20.382994 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:20.387665 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:20.387680 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:20.387697 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:20.387748 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:20.391472 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:20.392359 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:20.392370 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:20.392373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:20.392585 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:20.393055 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:20.393061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:20.393065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:20.393515 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:20.393858 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:20.394283 (+ 425us) server_negotiation.cc:299] Negotiation successful
0923 20:16:20.394612 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":298,"threads_started":1}
I20260923 20:16:20.396363 14199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:20.482048 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 14076
I20260923 20:16:20.498054 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 13716
I20260923 20:16:20.499212 13778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:20.605085 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 13716
I20260923 20:16:20.626228 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:43205
--webserver_interface=127.5.83.126
--webserver_port=39801
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:43205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:20.968691 14222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:20.968974 14222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:20.969063 14222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:20.976105 14222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:16:20.976195 14222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:20.976239 14222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:20.976295 14222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:16:20.976338 14222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:16:20.985849 14222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:43205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:43205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=39801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:20.987054 14222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:20.988221 14222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:20.999805 14230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:20.999840 14228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:20.999730 14227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:21.001027 14222 server_base.cc:1034] running on GCE node
I20260923 20:16:21.001742 14222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:21.003069 14222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:21.004272 14222 hybrid_clock.cc:648] HybridClock initialized: now 1790194581004216 us; error 48 us; skew 500 ppm
I20260923 20:16:21.004691 14222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:21 dist-test-slave-vl24 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194581, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:21.007753 14222 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:16:21.009389 14222 webserver.cc:460] Webserver started at http://127.5.83.126:39801/ using document root <none> and password file <none>
I20260923 20:16:21.010109 14222 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:21.010258 14222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:21.015676 14222 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 20:16:21.018656 14237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:21.020000 14222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:16:21.020136 14222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "1d20c83d429f4a278d1246ae5a71dfb3"
format_stamp: "Formatted at 2026-09-23 20:16:18 on dist-test-slave-vl24"
server_key: "6b6cd93c6c1cb356a0625c16acaec4db"
server_key_iv: "ee072485dbb4cda2e054b19dcc9e469f"
server_key_version: "encryptionkey@0"
I20260923 20:16:21.021155 14222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:21.052871 14222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:21.057428 14222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:21.057686 14222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:21.076683 14222 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:43205
I20260923 20:16:21.076695 14288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:43205 every 8 connection(s)
I20260923 20:16:21.078205 14222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:16:21.079860 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 14222
I20260923 20:16:21.080891 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:32997
--local_ip_for_outbound_sockets=127.5.83.65
--tserver_master_addrs=127.5.83.126:43205
--webserver_port=32983
--webserver_interface=127.5.83.65
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.83.65
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:16:21.088281 14289 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:16:21.092350 14289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3: Bootstrap starting.
I20260923 20:16:21.112572 14289 log.cc:824] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:21.121551 14289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:16:21.122088 14289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3: Bootstrap complete.
I20260923 20:16:21.131669 14289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43205 } }
I20260923 20:16:21.132519 14289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d20c83d429f4a278d1246ae5a71dfb3, State: Initialized, Role: FOLLOWER
I20260923 20:16:21.133284 14289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43205 } }
I20260923 20:16:21.133529 14289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:16:21.133658 14289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:16:21.133882 14289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:16:21.137835 14289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43205 } }
I20260923 20:16:21.138345 14289 leader_election.cc:302] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d20c83d429f4a278d1246ae5a71dfb3; no voters:
I20260923 20:16:21.138975 14289 leader_election.cc:288] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:16:21.139117 14293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:16:21.140424 14293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [term 2 LEADER]: Becoming Leader. State: Replica: 1d20c83d429f4a278d1246ae5a71dfb3, State: Running, Role: LEADER
I20260923 20:16:21.141074 14293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43205 } }
I20260923 20:16:21.141269 14289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:16:21.143702 14295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d20c83d429f4a278d1246ae5a71dfb3. Latest consensus state: current_term: 2 leader_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43205 } } }
I20260923 20:16:21.144138 14295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:21.144740 14294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d20c83d429f4a278d1246ae5a71dfb3" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43205 } } }
I20260923 20:16:21.145174 14294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:21.145180 14300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:16:21.151271 14300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:16:21.152704 14300 catalog_manager.cc:1259] Loaded cluster ID: 24ad9a3cfc164f83854f1f4e56ee1093
I20260923 20:16:21.152848 14300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:16:21.156883 14300 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:16:21.158540 14300 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1d20c83d429f4a278d1246ae5a71dfb3: Loaded TSK: 0
I20260923 20:16:21.160115 14300 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:16:21.453177 14291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:21.453456 14291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:21.453606 14291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:21.461205 14291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:21.461319 14291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:21.461479 14291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:16:21.471551 14291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.65.keytab
--principal=oryx/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:32997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=32983
--tserver_master_addrs=127.5.83.126:43205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:21.472977 14291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:21.474261 14291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:21.489378 14315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:21.489359 14316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:21.492888 14318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:21.494272 14291 server_base.cc:1034] running on GCE node
I20260923 20:16:21.494915 14291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:21.495976 14291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:21.497193 14291 hybrid_clock.cc:648] HybridClock initialized: now 1790194581497149 us; error 36 us; skew 500 ppm
I20260923 20:16:21.497521 14291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:21 dist-test-slave-vl24 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194581, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:21.505878 14291 init.cc:377] Logged in from keytab as oryx/127.5.83.65@KRBTEST.COM (short username oryx)
I20260923 20:16:21.507683 14291 webserver.cc:460] Webserver started at http://127.5.83.65:32983/ using document root <none> and password file <none>
I20260923 20:16:21.508592 14291 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:21.508764 14291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:21.514284 14291 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:16:21.517148 14325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:21.518743 14291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:16:21.518909 14291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "769f91f86de24654a6eccff2c6002b30"
format_stamp: "Formatted at 2026-09-23 20:16:18 on dist-test-slave-vl24"
server_key: "ae6f33d1767acf4f959dbed974e460e1"
server_key_iv: "29dc6be491634d53f14648518b3ab53d"
server_key_version: "encryptionkey@0"
I20260923 20:16:21.519881 14291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:21.548378 14291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:21.553656 14291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:21.553954 14291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:21.554944 14291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:21.556273 14291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:21.556394 14291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:21.556565 14291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:21.556670 14291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260923 20:16:21.587141 14291 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:32997
I20260923 20:16:21.587198 14437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:32997 every 8 connection(s)
I20260923 20:16:21.588693 14291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:16:21.598838 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 14291
I20260923 20:16:21.599534 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:41605
--local_ip_for_outbound_sockets=127.5.83.66
--tserver_master_addrs=127.5.83.126:43205
--webserver_port=41695
--webserver_interface=127.5.83.66
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.83.66
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:16:21 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:21 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:21.609189 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:21.591745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:51941)
0923 20:16:21.593007 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:21.593050 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:21.594739 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:21.596097 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:21.596108 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:21.596791 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:21.597326 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:21.597343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:21.599376 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:21.599383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:21.601086 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:21.601101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:21.602665 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:21.602671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:21.602917 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:21.603704 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:21.603729 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:21.607852 (+ 4123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1038,"thread_start_us":848,"threads_started":1}
W20260923 20:16:21.609424 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:21.591737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:51941 (local address 127.5.83.126:43205)
0923 20:16:21.592496 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:21.592507 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:21.593208 (+ 701us) server_negotiation.cc:407] Connection header received
0923 20:16:21.595271 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:21.595285 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:21.595607 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:21.595945 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:21.597518 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:21.599174 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:21.601247 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:21.602407 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:21.602630 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:21.608228 (+ 5598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:51941: BlockingRecv error: recv got EOF from 127.5.83.65:51941 (error 108)
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":282,"threads_started":1}
W20260923 20:16:21.610337 14438 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:43205 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.83.126:43205: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Sep 23 20:16:21 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:21 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:21.618849 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:21.610991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:39999)
0923 20:16:21.611179 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:21.611195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:21.611298 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:21.611640 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:21.611644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:21.612007 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:21.612143 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:21.612151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:21.613258 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:21.613264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:21.614178 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:21.614188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:21.614895 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:21.614901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:21.615076 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:21.615666 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:21.615683 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:21.618568 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 20:16:21.618988 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:21.611211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:39999 (local address 127.5.83.126:43205)
0923 20:16:21.611400 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:21.611405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:21.611425 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:16:21.611466 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:21.611470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:21.611521 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:21.611617 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:21.612277 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:21.613125 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:21.614336 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:21.614754 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:21.614875 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:21.618764 (+ 3889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:39999: BlockingRecv error: recv got EOF from 127.5.83.65:39999 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 23 20:16:21 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:21 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:21.628906 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:21.619598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:47327)
0923 20:16:21.619826 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:21.619849 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:21.619968 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:21.620311 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:21.620320 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:21.620845 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:21.621083 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:21.621100 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:21.622079 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:21.622086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:21.624048 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:21.624068 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:21.624815 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:21.624820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:21.625049 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:21.625710 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:21.625727 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:21.628595 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260923 20:16:21.629240 14438 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:16:21.629441 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:21.619733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:47327 (local address 127.5.83.126:43205)
0923 20:16:21.619881 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:21.619885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:21.619900 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:16:21.620096 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:21.620100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:21.620167 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:21.620241 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:21.621241 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:21.621925 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:21.624194 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:21.624679 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:21.624805 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:21.628800 (+ 3995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:47327: BlockingRecv error: recv got EOF from 127.5.83.65:47327 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260923 20:16:21.942854 14442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:21.943260 14442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:21.943454 14442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:21.952041 14442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:21.952229 14442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:21.952446 14442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:16:21.963685 14442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.66.keytab
--principal=oryx/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:41605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=41695
--tserver_master_addrs=127.5.83.126:43205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:21.965291 14442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:21.966703 14442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:21.981758 14447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:21.982136 14448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:21.982620 14450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:21.986280 14442 server_base.cc:1034] running on GCE node
I20260923 20:16:21.987205 14442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:21.988572 14442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:21.989794 14442 hybrid_clock.cc:648] HybridClock initialized: now 1790194581989744 us; error 50 us; skew 500 ppm
I20260923 20:16:21.990167 14442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:21 dist-test-slave-vl24 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194581, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:21.993410 14442 init.cc:377] Logged in from keytab as oryx/127.5.83.66@KRBTEST.COM (short username oryx)
I20260923 20:16:21.994954 14442 webserver.cc:460] Webserver started at http://127.5.83.66:41695/ using document root <none> and password file <none>
I20260923 20:16:21.995735 14442 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:21.995889 14442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:22.001701 14442 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:16:22.004698 14457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:22.006135 14442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:16:22.006328 14442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "650bc7c075fc4d5bab67854cd27dfd89"
format_stamp: "Formatted at 2026-09-23 20:16:19 on dist-test-slave-vl24"
server_key: "9456e3dbb7749545f2ec5aec74b75418"
server_key_iv: "e21d6af4c23a433b89bba6b19d8d4d1b"
server_key_version: "encryptionkey@0"
I20260923 20:16:22.007387 14442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:22.020445 14442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:22.025368 14442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:22.025702 14442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:22.026798 14442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:22.028211 14442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:22.028342 14442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:22.028461 14442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:22.028585 14442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:22.062522 14442 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:41605
I20260923 20:16:22.062587 14569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:41605 every 8 connection(s)
I20260923 20:16:22.063995 14442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:16:22.067118 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 14442
I20260923 20:16:22.067870 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:36453
--local_ip_for_outbound_sockets=127.5.83.67
--tserver_master_addrs=127.5.83.126:43205
--webserver_port=40641
--webserver_interface=127.5.83.67
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.83.67
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:16:22.082856 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:22.066847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:39263 (local address 127.5.83.126:43205)
0923 20:16:22.067057 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:22.067062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:22.068614 (+ 1552us) server_negotiation.cc:407] Connection header received
0923 20:16:22.070770 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:22.070777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:22.070889 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:22.071012 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:22.072872 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.073678 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.076253 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.077003 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.077149 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.082602 (+ 5453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:39263: BlockingRecv error: recv got EOF from 127.5.83.66:39263 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:16:22.083658 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:22.067533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:39263)
0923 20:16:22.068411 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:22.068456 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:22.070451 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:22.071297 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:22.071307 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:22.071902 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:22.072658 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.072682 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.073881 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.073886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.076004 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.076023 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.077232 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.077237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.077505 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.078252 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:22.078278 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:22.082375 (+ 4097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":361,"threads_started":1}
W20260923 20:16:22.084362 14570 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:43205 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.83.126:43205: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:22.093039 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:22.085018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:57913)
0923 20:16:22.085257 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:22.085277 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:22.085395 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:22.085722 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:22.085726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:22.086081 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:22.086223 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.086231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.087283 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.087289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.088208 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.088219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.089043 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.089049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.089226 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.089820 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:22.089838 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:22.092732 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130}
W20260923 20:16:22.093222 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:22.085082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:57913 (local address 127.5.83.126:43205)
0923 20:16:22.085251 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:22.085256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:22.085347 (+ 91us) server_negotiation.cc:407] Connection header received
0923 20:16:22.085529 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:22.085539 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:22.085600 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:22.085699 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:22.086358 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.087132 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.088349 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.088871 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.089031 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.092945 (+ 3914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:57913: BlockingRecv error: recv got EOF from 127.5.83.66:57913 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:22.102011 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:22.093907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:40819)
0923 20:16:22.094075 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:22.094093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:22.094214 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:22.094747 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:22.094751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:22.095072 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:22.095213 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.095220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.096262 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.096268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.097254 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.097266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.097981 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.097987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.098169 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.098752 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:22.098770 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:22.101706 (+ 2936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 20:16:22.102125 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:22.094348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:40819 (local address 127.5.83.126:43205)
0923 20:16:22.094506 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:22.094509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:22.094523 (+ 14us) server_negotiation.cc:407] Connection header received
0923 20:16:22.094564 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:22.094567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:22.094619 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:22.094720 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:22.095351 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.096131 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.097396 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.097824 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.097951 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.101926 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:40819: BlockingRecv error: recv got EOF from 127.5.83.66:40819 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260923 20:16:22.102456 14570 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:16:22.412752 14573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:22.413074 14573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:22.413244 14573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:22.421058 14573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:22.421198 14573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:22.421412 14573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:16:22.431969 14573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/krb5kdc/oryx_127.5.83.67.keytab
--principal=oryx/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:36453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=40641
--tserver_master_addrs=127.5.83.126:43205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:22.433455 14573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:22.434814 14573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:22.449263 14578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:22.449440 14579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:22.449977 14581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:22.450466 14573 server_base.cc:1034] running on GCE node
I20260923 20:16:22.451177 14573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:22.452201 14573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:22.453452 14573 hybrid_clock.cc:648] HybridClock initialized: now 1790194582453393 us; error 50 us; skew 500 ppm
I20260923 20:16:22.453837 14573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194582, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:22.457230 14573 init.cc:377] Logged in from keytab as oryx/127.5.83.67@KRBTEST.COM (short username oryx)
I20260923 20:16:22.458696 14573 webserver.cc:460] Webserver started at http://127.5.83.67:40641/ using document root <none> and password file <none>
I20260923 20:16:22.459369 14573 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:22.459515 14573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:22.464905 14573 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:16:22.467551 14588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:22.468873 14573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:16:22.469043 14573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "af4f91afcfe74a0bbd056b498b9a892c"
format_stamp: "Formatted at 2026-09-23 20:16:19 on dist-test-slave-vl24"
server_key: "d056f361b4ad06d8990712ad6579c88a"
server_key_iv: "358963b58e1b6d85b8e22bc9d7644dec"
server_key_version: "encryptionkey@0"
I20260923 20:16:22.469986 14573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:22.484930 14573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:22.488865 14573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:22.489127 14573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:22.490051 14573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:22.491345 14573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:22.491459 14573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:22.491581 14573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:22.491667 14573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:22.524144 14573 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:36453
I20260923 20:16:22.524230 14700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:36453 every 8 connection(s)
I20260923 20:16:22.525718 14573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:16:22.527868 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 14573
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 20:16:22.546113 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:22.528268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:46117 (local address 127.5.83.126:43205)
0923 20:16:22.528478 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:22.528484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:22.533208 (+ 4724us) server_negotiation.cc:407] Connection header received
0923 20:16:22.535300 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:22.535305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:22.535419 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:22.535533 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:22.537234 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.537955 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.539476 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.539897 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.540021 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.545800 (+ 5779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:46117: BlockingRecv error: recv got EOF from 127.5.83.67:46117 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":70}
I20260923 20:16:22.546866 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:22.528880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:46117)
0923 20:16:22.533045 (+ 4165us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:22.533085 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:22.535038 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:22.535775 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:22.535784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:22.536264 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:22.537078 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.537096 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.538111 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.538122 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.539338 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.539349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.540124 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.540132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.540348 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.541032 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:22.541061 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:22.545613 (+ 4552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3910,"thread_start_us":3729,"threads_started":1}
W20260923 20:16:22.547696 14701 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:43205 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.83.126:43205: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
I20260923 20:16:22.548220 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:22.528873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.5.83.126:43205)
0923 20:16:22.533578 (+ 4705us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:22.533583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:22.533601 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:16:22.533664 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:22.533672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:22.533749 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:22.533871 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:22.535440 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.536225 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.537680 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.538129 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.538261 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.540980 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:22.540991 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:22.541011 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:22.541097 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:22.544941 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:22.545591 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:22.545599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:22.545606 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:22.545765 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:22.546134 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:22.546139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:22.546142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:22.546714 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:22.547090 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:22.547503 (+ 413us) server_negotiation.cc:299] Negotiation successful
0923 20:16:22.547866 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4591,"thread_start_us":4476,"threads_started":1}
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:22.557071 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:22.548565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:36587)
0923 20:16:22.548753 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:22.548771 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:22.548886 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:22.549161 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:22.549166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:22.549580 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:22.549715 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.549723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.551012 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.551016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.552047 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.552057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.552929 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.552934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.553121 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.553734 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:22.553756 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:22.556859 (+ 3103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 20:16:22.557331 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:22.548415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:36587 (local address 127.5.83.126:43205)
0923 20:16:22.548597 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:22.548602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:22.548838 (+ 236us) server_negotiation.cc:407] Connection header received
0923 20:16:22.548984 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:22.548988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:22.549038 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:22.549116 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:22.549839 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.550864 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.552173 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.552781 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.552930 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.557077 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:36587: BlockingRecv error: recv got EOF from 127.5.83.67:36587 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:22.565290 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:22.557855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:52769)
0923 20:16:22.558005 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:22.558020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:22.558117 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:22.558388 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:22.558390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:22.558656 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:22.558797 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.558805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.559737 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.559740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.560696 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.560706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.561300 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.561303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.561447 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.561931 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:22.561947 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:22.565084 (+ 3137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 20:16:22.565681 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:22.557941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:52769 (local address 127.5.83.126:43205)
0923 20:16:22.558166 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:22.558169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:22.558184 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:16:22.558226 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:22.558229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:22.558282 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:22.558357 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:22.558911 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.559627 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.560814 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.561192 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.561277 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.565289 (+ 4012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:52769: BlockingRecv error: recv got EOF from 127.5.83.67:52769 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260923 20:16:22.565761 14701 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:22 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:22.638223 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:22.629788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:60165)
0923 20:16:22.630289 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:22.630306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:22.630419 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:22.630705 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:22.630708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:22.631133 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:22.631319 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.631328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.632262 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.632266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.633237 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:22.633248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.633966 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.633970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:22.634090 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.634680 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:22.634698 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:22.637886 (+ 3188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":296,"threads_started":1}
W20260923 20:16:22.638345 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:22.629890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:60165 (local address 127.5.83.126:43205)
0923 20:16:22.630124 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:22.630130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:22.630383 (+ 253us) server_negotiation.cc:407] Connection header received
0923 20:16:22.630521 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:22.630525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:22.630586 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:22.630665 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:22.631432 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.632143 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.633362 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:22.633794 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:22.633895 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:22.638118 (+ 4223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:60165: BlockingRecv error: recv got EOF from 127.5.83.65:60165 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 23 20:16:23 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:23 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:23.112039 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:23.103044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:40433)
0923 20:16:23.103637 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:23.103657 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:23.103823 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:23.104195 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:23.104199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:23.104762 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:23.104956 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:23.104965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:23.105978 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:23.105983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:23.106988 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:23.107001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:23.107842 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:23.107846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:23.108027 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:23.108671 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:23.108689 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:23.111687 (+ 2998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":299,"threads_started":1}
W20260923 20:16:23.112229 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:23.103229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:40433 (local address 127.5.83.126:43205)
0923 20:16:23.103497 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:23.103505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:23.103746 (+ 241us) server_negotiation.cc:407] Connection header received
0923 20:16:23.103954 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:23.103959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:23.104031 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:23.104167 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:23.105108 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:23.105831 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:23.107195 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:23.107665 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:23.107815 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:23.111938 (+ 4123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:40433: BlockingRecv error: recv got EOF from 127.5.83.66:40433 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 23 20:16:23 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:23 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:23.575212 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:23.566389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:56005)
0923 20:16:23.566941 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:23.566957 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:23.567084 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:23.567467 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:23.567472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:23.568074 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:23.568362 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:23.568374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:23.569724 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:23.569729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:23.570661 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:23.570672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:23.571321 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:23.571325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:23.571437 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:23.571969 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:23.571986 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:23.574979 (+ 2993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":295,"threads_started":1}
W20260923 20:16:23.575426 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:23.566463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:56005 (local address 127.5.83.126:43205)
0923 20:16:23.566634 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:23.566638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:23.567039 (+ 401us) server_negotiation.cc:407] Connection header received
0923 20:16:23.567203 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:23.567208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:23.567291 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:23.567407 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:23.568567 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:23.569589 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:23.570775 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:23.571173 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:23.571270 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:23.575176 (+ 3906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:56005: BlockingRecv error: recv got EOF from 127.5.83.67:56005 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 20:16:23 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:23 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:23.647835 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:23.639182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:54505)
0923 20:16:23.639753 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:23.639772 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:23.639912 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:23.640381 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:23.640385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:23.640994 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:23.641183 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:23.641196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:23.642195 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:23.642202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:23.643106 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:23.643117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:23.643796 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:23.643801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:23.643964 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:23.644658 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:23.644676 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:23.647478 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":335,"threads_started":1}
W20260923 20:16:23.648140 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:23.639307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:54505 (local address 127.5.83.126:43205)
0923 20:16:23.639499 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:23.639506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:23.639864 (+ 358us) server_negotiation.cc:407] Connection header received
0923 20:16:23.640050 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:23.640062 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:23.640161 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:23.640296 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:23.641303 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:23.642062 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:23.643243 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:23.643642 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:23.643785 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:23.647686 (+ 3901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:54505: BlockingRecv error: recv got EOF from 127.5.83.65:54505 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 20:16:24 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:24 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:24.121564 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:24.113136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:58715)
0923 20:16:24.113716 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:24.113732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:24.113853 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:24.114183 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:24.114186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:24.114587 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:24.114741 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:24.114751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:24.115756 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:24.115760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:24.116921 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:24.116933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:24.117870 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:24.117874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:24.117985 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:24.118583 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:24.118602 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:24.121330 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":305,"threads_started":1}
W20260923 20:16:24.121825 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:24.113290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:58715 (local address 127.5.83.126:43205)
0923 20:16:24.113571 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:24.113579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:24.113805 (+ 226us) server_negotiation.cc:407] Connection header received
0923 20:16:24.113969 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:24.113973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:24.114038 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:24.114126 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:24.114882 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:24.115624 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:24.117063 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:24.117703 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:24.117827 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:24.121554 (+ 3727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:58715: BlockingRecv error: recv got EOF from 127.5.83.66:58715 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 23 20:16:24 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:24 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:24.584965 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:24.576073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:59419)
0923 20:16:24.576623 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:24.576651 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:24.576767 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:24.577197 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:24.577200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:24.577753 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:24.577969 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:24.577982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:24.579129 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:24.579133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:24.580004 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:24.580014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:24.580752 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:24.580758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:24.580964 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:24.581522 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:24.581540 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:24.584723 (+ 3183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":300,"threads_started":1}
W20260923 20:16:24.585168 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:24.576271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:59419 (local address 127.5.83.126:43205)
0923 20:16:24.576435 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:24.576439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:24.576736 (+ 297us) server_negotiation.cc:407] Connection header received
0923 20:16:24.576903 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:24.576909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:24.577007 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:24.577155 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:24.578120 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:24.578993 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:24.580126 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:24.580571 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:24.580720 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:24.584899 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:59419: BlockingRecv error: recv got EOF from 127.5.83.67:59419 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 20:16:24 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:24 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:24.657460 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:24.648913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:43051)
0923 20:16:24.649397 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:24.649414 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:24.649535 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:24.649920 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:24.649923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:24.650387 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:24.650593 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:24.650605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:24.652032 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:24.652036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:24.653037 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:24.653049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:24.653664 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:24.653667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:24.653796 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:24.654421 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:24.654439 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:24.657217 (+ 2778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":274,"threads_started":1}
W20260923 20:16:24.657671 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:24.648970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:43051 (local address 127.5.83.126:43205)
0923 20:16:24.649251 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:24.649258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:24.649534 (+ 276us) server_negotiation.cc:407] Connection header received
0923 20:16:24.649634 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:24.649638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:24.649767 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:24.649957 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:24.650814 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:24.651886 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:24.653157 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:24.653531 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:24.653626 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:24.657441 (+ 3815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:43051: BlockingRecv error: recv got EOF from 127.5.83.65:43051 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 23 20:16:25 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:25 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:25.132313 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:25.122488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:59641)
0923 20:16:25.123003 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:25.123020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:25.123167 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:25.123492 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:25.123495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:25.123846 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:25.124033 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:25.124043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:25.125338 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:25.125342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:25.126682 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:25.126697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:25.127307 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:25.127311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:25.127435 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:25.128138 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:25.128157 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:25.131899 (+ 3742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":275,"threads_started":1}
W20260923 20:16:25.133132 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:25.122627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:59641 (local address 127.5.83.126:43205)
0923 20:16:25.122869 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:25.122876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:25.123125 (+ 249us) server_negotiation.cc:407] Connection header received
0923 20:16:25.123266 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:25.123270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:25.123334 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:25.123433 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:25.124181 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:25.125203 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:25.126795 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:25.127187 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:25.127278 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:25.132458 (+ 5180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:59641: BlockingRecv error: recv got EOF from 127.5.83.66:59641 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 23 20:16:25 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:25 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:25.594964 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:25.586119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:47005)
0923 20:16:25.586820 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:25.586836 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:25.586991 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:25.587366 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:25.587370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:25.587840 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:25.588092 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:25.588104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:25.589197 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:25.589203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:25.590441 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:25.590453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:25.591146 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:25.591150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:25.591278 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:25.591857 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:25.591883 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:25.594712 (+ 2829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":459,"threads_started":1}
W20260923 20:16:25.595165 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:25.586151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:47005 (local address 127.5.83.126:43205)
0923 20:16:25.586386 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:25.586397 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:25.586935 (+ 538us) server_negotiation.cc:407] Connection header received
0923 20:16:25.587124 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:25.587133 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:25.587225 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:25.587311 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:25.588228 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:25.589067 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:25.590572 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:25.591011 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:25.591112 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:25.594921 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:47005: BlockingRecv error: recv got EOF from 127.5.83.67:47005 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 23 20:16:25 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:25 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:25.666567 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:25.658521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:57713)
0923 20:16:25.659059 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:25.659097 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:25.659246 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:25.659578 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:25.659582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:25.659945 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:25.660124 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:25.660133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:25.661187 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:25.661191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:25.662144 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:25.662156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:25.662759 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:25.662763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:25.662890 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:25.663539 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:25.663557 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:25.666336 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":292,"threads_started":1}
W20260923 20:16:25.666862 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:25.658591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:57713 (local address 127.5.83.126:43205)
0923 20:16:25.658808 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:25.658813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:25.659179 (+ 366us) server_negotiation.cc:407] Connection header received
0923 20:16:25.659337 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:25.659341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:25.659420 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:25.659500 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:25.660246 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:25.661055 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:25.662260 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:25.662635 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:25.662738 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:25.666559 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:57713: BlockingRecv error: recv got EOF from 127.5.83.65:57713 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 23 20:16:26 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:26 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:26.142457 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:26.134475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:42775)
0923 20:16:26.134996 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:26.135014 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:26.135126 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:26.135475 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:26.135478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:26.135818 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:26.135970 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:26.135979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:26.136945 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:26.136948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:26.137884 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:26.137895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:26.138574 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:26.138578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:26.138701 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:26.139256 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:26.139274 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:26.142206 (+ 2932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":288,"threads_started":1}
W20260923 20:16:26.142717 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:26.134536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:42775 (local address 127.5.83.126:43205)
0923 20:16:26.134808 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:26.134812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:26.135115 (+ 303us) server_negotiation.cc:407] Connection header received
0923 20:16:26.135222 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:26.135226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:26.135315 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:26.135401 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:26.136090 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:26.136839 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:26.138009 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:26.138448 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:26.138553 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:26.142436 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:42775: BlockingRecv error: recv got EOF from 127.5.83.66:42775 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Sep 23 20:16:26 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:26 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:26.609661 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:26.596077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:50335)
0923 20:16:26.596622 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:26.596652 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:26.596812 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:26.597191 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:26.597195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:26.597556 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:26.597694 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:26.597702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:26.598896 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:26.598900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:26.599865 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:26.599876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:26.600701 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:26.600707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:26.600923 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:26.606222 (+ 5299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:26.606243 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:26.609253 (+ 3010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":271,"threads_started":1}
W20260923 20:16:26.609786 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:26.596146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:50335 (local address 127.5.83.126:43205)
0923 20:16:26.596398 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:26.596406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:26.596748 (+ 342us) server_negotiation.cc:407] Connection header received
0923 20:16:26.596956 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:26.596962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:26.597065 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:26.597150 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:26.597811 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:26.598753 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:26.600031 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:26.600477 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:26.600693 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:26.609494 (+ 8801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:50335: BlockingRecv error: recv got EOF from 127.5.83.67:50335 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 23 20:16:26 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:26 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:26.676386 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:26.667649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:44781)
0923 20:16:26.668187 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:26.668203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:26.668360 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:26.668813 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:26.668817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:26.669260 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:26.669410 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:26.669419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:26.670399 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:26.670406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:26.671443 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:26.671477 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:26.672359 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:26.672366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:26.672603 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:26.673262 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:26.673280 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:26.676145 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":261,"threads_started":1}
W20260923 20:16:26.676642 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:26.667733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:44781 (local address 127.5.83.126:43205)
0923 20:16:26.667981 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:26.667987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:26.668296 (+ 309us) server_negotiation.cc:407] Connection header received
0923 20:16:26.668485 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:26.668493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:26.668645 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:26.668782 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:26.669565 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:26.670268 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:26.671602 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:26.672193 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:26.672318 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:26.676355 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:44781: BlockingRecv error: recv got EOF from 127.5.83.65:44781 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 23 20:16:27 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:27 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:27.152004 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:27.143570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:56135)
0923 20:16:27.144164 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:27.144180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:27.144318 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:27.144728 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:27.144734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:27.145295 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:27.145476 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:27.145486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:27.146490 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:27.146494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:27.147476 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:27.147488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:27.148227 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:27.148231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:27.148355 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:27.149035 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:27.149053 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:27.151729 (+ 2676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":239,"threads_started":1}
W20260923 20:16:27.152168 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:27.143808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:56135 (local address 127.5.83.126:43205)
0923 20:16:27.144058 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:27.144066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:27.144259 (+ 193us) server_negotiation.cc:407] Connection header received
0923 20:16:27.144452 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:27.144459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:27.144564 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:27.144664 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:27.145631 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:27.146363 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:27.147645 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:27.148099 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:27.148190 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:27.151920 (+ 3730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:56135: BlockingRecv error: recv got EOF from 127.5.83.66:56135 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 23 20:16:27 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:27 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:27.619095 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:27.610719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:54011)
0923 20:16:27.611294 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:27.611311 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:27.611418 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:27.611756 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:27.611759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:27.612179 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:27.612337 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:27.612345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:27.613394 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:27.613399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:27.614296 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:27.614307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:27.614964 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:27.614968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:27.615102 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:27.615661 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:27.615677 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:27.618857 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":340,"threads_started":1}
W20260923 20:16:27.619272 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:27.610834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:54011 (local address 127.5.83.126:43205)
0923 20:16:27.610997 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:27.611034 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:27.611383 (+ 349us) server_negotiation.cc:407] Connection header received
0923 20:16:27.611560 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:27.611563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:27.611625 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:27.611711 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:27.612463 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:27.613254 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:27.614408 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:27.614833 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:27.614923 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:27.619047 (+ 4124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:54011: BlockingRecv error: recv got EOF from 127.5.83.67:54011 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 20:16:27 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:27 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:27.686173 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:27.677461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:53405)
0923 20:16:27.678043 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:27.678062 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:27.678221 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:27.678643 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:27.678648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:27.679198 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:27.679439 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:27.679455 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:27.680447 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:27.680450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:27.681370 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:27.681381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:27.682019 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:27.682023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:27.682143 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:27.682847 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:27.682869 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:27.685944 (+ 3075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":319,"threads_started":1}
W20260923 20:16:27.686425 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:27.677470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:53405 (local address 127.5.83.126:43205)
0923 20:16:27.677674 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:27.677681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:27.678150 (+ 469us) server_negotiation.cc:407] Connection header received
0923 20:16:27.678357 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:27.678362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:27.678444 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:27.678546 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:27.679596 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:27.680326 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:27.681493 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:27.681883 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:27.681980 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:27.686173 (+ 4193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:53405: BlockingRecv error: recv got EOF from 127.5.83.65:53405 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 20:16:28 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:28 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:28.161252 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:28.153003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:34427)
0923 20:16:28.153552 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:28.153568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:28.153679 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:28.153954 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:28.153957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:28.154333 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:28.154477 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:28.154487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:28.155464 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:28.155468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:28.156393 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:28.156405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:28.157230 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:28.157235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:28.157376 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:28.158046 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:28.158064 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:28.161033 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":321,"threads_started":1}
W20260923 20:16:28.161515 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:28.153088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:34427 (local address 127.5.83.126:43205)
0923 20:16:28.153253 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:28.153257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:28.153641 (+ 384us) server_negotiation.cc:407] Connection header received
0923 20:16:28.153771 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:28.153775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:28.153834 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:28.153936 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:28.154622 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:28.155344 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:28.156567 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:28.157041 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:28.157155 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:28.161268 (+ 4113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:34427: BlockingRecv error: recv got EOF from 127.5.83.66:34427 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 20:16:28 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:28 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:28.629451 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:28.620264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:34893)
0923 20:16:28.620833 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:28.620850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:28.621007 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:28.621319 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:28.621323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:28.621706 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:28.621900 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:28.621913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:28.623331 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:28.623339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:28.624520 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:28.624585 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:28.625453 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:28.625459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:28.625599 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:28.626248 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:28.626268 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:28.629208 (+ 2940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":323,"threads_started":1}
W20260923 20:16:28.629685 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:28.620516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:34893 (local address 127.5.83.126:43205)
0923 20:16:28.620773 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:28.620780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:28.620958 (+ 178us) server_negotiation.cc:407] Connection header received
0923 20:16:28.621119 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:28.621123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:28.621189 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:28.621277 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:28.622070 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:28.623131 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:28.624732 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:28.625247 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:28.625364 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:28.629423 (+ 4059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:34893: BlockingRecv error: recv got EOF from 127.5.83.67:34893 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 23 20:16:28 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:28 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:28.694882 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:28.687105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:39707)
0923 20:16:28.687625 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:28.687642 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:28.687762 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:28.688079 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:28.688082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:28.688485 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:28.688676 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:28.688685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:28.689724 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:28.689727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:28.690626 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:28.690636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:28.691265 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:28.691269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:28.691379 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:28.691924 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:28.691941 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:28.694622 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":264,"threads_started":1}
W20260923 20:16:28.695181 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:28.687253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:39707 (local address 127.5.83.126:43205)
0923 20:16:28.687479 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:28.687485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:28.687747 (+ 262us) server_negotiation.cc:407] Connection header received
0923 20:16:28.687863 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:28.687867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:28.687930 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:28.688010 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:28.688808 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:28.689606 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:28.690743 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:28.691127 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:28.691226 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:28.694882 (+ 3656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:39707: BlockingRecv error: recv got EOF from 127.5.83.65:39707 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 23 20:16:29 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:29 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:29.174450 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:29.162618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:33595)
0923 20:16:29.163411 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:29.163434 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:29.163589 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:29.164160 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:29.164169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:29.165171 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:29.165454 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:29.165470 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:29.167052 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:29.167057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:29.168214 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:29.168231 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:29.169550 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:29.169560 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:29.169726 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:29.170450 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:29.170474 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:29.173870 (+ 3396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":351,"threads_started":1}
W20260923 20:16:29.174520 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:29.162441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:33595 (local address 127.5.83.126:43205)
0923 20:16:29.162639 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:29.162646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:29.163515 (+ 869us) server_negotiation.cc:407] Connection header received
0923 20:16:29.163807 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:29.163813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:29.163926 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:29.164030 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:29.165683 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:29.166853 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:29.168396 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:29.169347 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:29.169526 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:29.174105 (+ 4579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:33595: BlockingRecv error: recv got EOF from 127.5.83.66:33595 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 20:16:29 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:29 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:29.639303 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:29.630525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:49761)
0923 20:16:29.631182 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:29.631198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:29.631342 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:29.631691 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:29.631697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:29.632235 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:29.632407 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:29.632416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:29.633480 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:29.633485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:29.634435 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:29.634450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:29.635268 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:29.635275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:29.635416 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:29.636008 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:29.636027 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:29.638962 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":325,"threads_started":1}
W20260923 20:16:29.639429 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:29.630587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:49761 (local address 127.5.83.126:43205)
0923 20:16:29.630775 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:29.630781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:29.631291 (+ 510us) server_negotiation.cc:407] Connection header received
0923 20:16:29.631462 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:29.631466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:29.631529 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:29.631615 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:29.632590 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:29.633337 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:29.634592 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:29.635086 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:29.635222 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:29.639186 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:49761: BlockingRecv error: recv got EOF from 127.5.83.67:49761 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 23 20:16:29 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:29 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:29.704275 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:29.695921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:40327)
0923 20:16:29.696426 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:29.696442 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:29.696595 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:29.696891 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:29.696894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:29.697315 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:29.697496 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:29.697505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:29.698438 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:29.698441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:29.699374 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:29.699385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:29.700107 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:29.700112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:29.700222 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:29.700955 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:29.700974 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:29.703907 (+ 2933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":249,"threads_started":1}
W20260923 20:16:29.704417 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:29.696185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:40327 (local address 127.5.83.126:43205)
0923 20:16:29.696405 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:29.696409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:29.696579 (+ 170us) server_negotiation.cc:407] Connection header received
0923 20:16:29.696707 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:29.696711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:29.696772 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:29.696851 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:29.697616 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:29.698319 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:29.699505 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:29.699946 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:29.700040 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:29.704126 (+ 4086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:40327: BlockingRecv error: recv got EOF from 127.5.83.65:40327 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 23 20:16:30 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:30 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:30.184821 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:30.175421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:52615)
0923 20:16:30.176001 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:30.176019 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:30.176152 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:30.176493 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:30.176496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:30.176998 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:30.177159 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:30.177168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:30.178491 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:30.178495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:30.179462 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:30.179475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:30.180373 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:30.180381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:30.180607 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:30.181359 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:30.181392 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:30.184512 (+ 3120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":312,"threads_started":1}
W20260923 20:16:30.185091 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:30.175598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:52615 (local address 127.5.83.126:43205)
0923 20:16:30.175776 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:30.175781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:30.176137 (+ 356us) server_negotiation.cc:407] Connection header received
0923 20:16:30.176253 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:30.176257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:30.176337 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:30.176454 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:30.177313 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:30.178323 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:30.179628 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:30.180110 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:30.180248 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:30.184777 (+ 4529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:52615: BlockingRecv error: recv got EOF from 127.5.83.66:52615 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 20:16:30 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:30 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:30.649258 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:30.640426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:48577)
0923 20:16:30.641047 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:30.641063 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:30.641178 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:30.641599 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:30.641602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:30.642063 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:30.642228 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:30.642237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:30.643420 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:30.643425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:30.644351 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:30.644361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:30.645118 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:30.645123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:30.645242 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:30.645830 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:30.645847 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:30.649024 (+ 3177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":332,"threads_started":1}
W20260923 20:16:30.649475 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:30.640648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:48577 (local address 127.5.83.126:43205)
0923 20:16:30.640840 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:30.640844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:30.641166 (+ 322us) server_negotiation.cc:407] Connection header received
0923 20:16:30.641296 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:30.641300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:30.641433 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:30.641562 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:30.642388 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:30.643282 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:30.644484 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:30.644944 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:30.645053 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:30.649198 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:48577: BlockingRecv error: recv got EOF from 127.5.83.67:48577 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 20:16:30 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:30 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:30.714650 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:30.705405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:42041)
0923 20:16:30.705992 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:30.706010 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:30.706143 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:30.706492 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:30.706496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:30.706894 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:30.707064 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:30.707073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:30.708426 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:30.708431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:30.709761 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:30.709774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:30.710491 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:30.710496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:30.710622 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:30.711261 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:30.711284 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:30.714388 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":339,"threads_started":1}
W20260923 20:16:30.714927 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:30.705608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:42041 (local address 127.5.83.126:43205)
0923 20:16:30.705815 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:30.705819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:30.706088 (+ 269us) server_negotiation.cc:407] Connection header received
0923 20:16:30.706248 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:30.706252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:30.706315 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:30.706408 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:30.707216 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:30.708270 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:30.709888 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:30.710328 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:30.710432 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:30.714647 (+ 4215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:42041: BlockingRecv error: recv got EOF from 127.5.83.65:42041 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 20:16:31 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:31 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:31.194833 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:31.185851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:39009)
0923 20:16:31.186655 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:31.186672 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:31.186872 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:31.187255 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:31.187259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:31.187739 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:31.187904 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:31.187913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:31.189046 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:31.189051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:31.190034 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:31.190046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:31.190755 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:31.190760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:31.190896 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:31.191531 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:31.191550 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:31.194478 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":422,"threads_started":1}
W20260923 20:16:31.194967 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:31.185899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:39009 (local address 127.5.83.126:43205)
0923 20:16:31.186258 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:31.186268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:31.186758 (+ 490us) server_negotiation.cc:407] Connection header received
0923 20:16:31.187009 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:31.187015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:31.187097 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:31.187215 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:31.188035 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:31.188893 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:31.190177 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:31.190613 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:31.190710 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:31.194697 (+ 3987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:39009: BlockingRecv error: recv got EOF from 127.5.83.66:39009 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 23 20:16:31 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:31 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:31.662745 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:31.650248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:46763)
0923 20:16:31.650734 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:31.650750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:31.650864 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:31.651257 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:31.651261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:31.651668 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:31.651830 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:31.651838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:31.653002 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:31.653007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:31.653995 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:31.654006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:31.654668 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:31.654672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:31.654799 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:31.659129 (+ 4330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:31.659157 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:31.662367 (+ 3210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":279,"threads_started":1}
W20260923 20:16:31.662783 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:31.650319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:46763 (local address 127.5.83.126:43205)
0923 20:16:31.650560 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:31.650566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:31.650827 (+ 261us) server_negotiation.cc:407] Connection header received
0923 20:16:31.650989 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:31.650993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:31.651063 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:31.651135 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:31.651981 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:31.652858 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:31.654115 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:31.654533 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:31.654631 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:31.662560 (+ 7929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:46763: BlockingRecv error: recv got EOF from 127.5.83.67:46763 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 20:16:31 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:31 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:31.724398 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:31.715568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:36137)
0923 20:16:31.716069 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:31.716086 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:31.716208 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:31.716514 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:31.716518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:31.717147 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:31.717294 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:31.717303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:31.718301 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:31.718305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:31.719276 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:31.719288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:31.720072 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:31.720076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:31.720190 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:31.720929 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:31.720986 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:31.724104 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":307,"threads_started":1}
W20260923 20:16:31.724745 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:31.715649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:36137 (local address 127.5.83.126:43205)
0923 20:16:31.715911 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:31.715917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:31.716179 (+ 262us) server_negotiation.cc:407] Connection header received
0923 20:16:31.716311 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:31.716314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:31.716377 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:31.716460 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:31.717425 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:31.718174 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:31.719421 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:31.719897 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:31.720028 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:31.724398 (+ 4370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:36137: BlockingRecv error: recv got EOF from 127.5.83.65:36137 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 20:16:32 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:32 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:32.203965 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:32.195855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:35901)
0923 20:16:32.196395 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:32.196410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:32.196517 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:32.196876 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:32.196879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:32.197223 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:32.197362 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:32.197370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:32.198395 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:32.198399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:32.199354 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:32.199365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:32.200016 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:32.200020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:32.200134 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:32.200803 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:32.200821 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:32.203619 (+ 2798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":313,"threads_started":1}
W20260923 20:16:32.204061 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:32.195911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:35901 (local address 127.5.83.126:43205)
0923 20:16:32.196073 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:32.196078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:32.196513 (+ 435us) server_negotiation.cc:407] Connection header received
0923 20:16:32.196687 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:32.196691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:32.196754 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:32.196840 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:32.197515 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:32.198267 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:32.199481 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:32.199882 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:32.199977 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:32.203816 (+ 3839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:35901: BlockingRecv error: recv got EOF from 127.5.83.66:35901 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 20:16:32 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:32 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:32.672286 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:32.663810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:59251)
0923 20:16:32.664288 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:32.664304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:32.664424 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:32.664735 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:32.664738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:32.665093 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:32.665305 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:32.665319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:32.666784 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:32.666790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:32.667713 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:32.667724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:32.668360 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:32.668364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:32.668469 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:32.669132 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:32.669149 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:32.671948 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":290,"threads_started":1}
W20260923 20:16:32.672477 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:32.664020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:59251 (local address 127.5.83.126:43205)
0923 20:16:32.664287 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:32.664294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:32.664383 (+ 89us) server_negotiation.cc:407] Connection header received
0923 20:16:32.664542 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:32.664555 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:32.664616 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:32.664697 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:32.665479 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:32.666605 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:32.667841 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:32.668237 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:32.668330 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:32.672199 (+ 3869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:59251: BlockingRecv error: recv got EOF from 127.5.83.67:59251 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Sep 23 20:16:32 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:32 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:32.734324 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:32.725536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:35137)
0923 20:16:32.726335 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:32.726352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:32.726505 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:32.726855 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:32.726860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:32.727272 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:32.727439 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:32.727448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:32.728477 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:32.728484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:32.729440 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:32.729452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:32.730229 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:32.730236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:32.730403 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:32.730971 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:32.730987 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:32.734061 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":562,"threads_started":1}
W20260923 20:16:32.734467 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:32.725677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:35137 (local address 127.5.83.126:43205)
0923 20:16:32.725885 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:32.725890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:32.726442 (+ 552us) server_negotiation.cc:407] Connection header received
0923 20:16:32.726635 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:32.726641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:32.726723 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:32.726822 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:32.727585 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:32.728336 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:32.729609 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:32.730056 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:32.730197 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:32.734251 (+ 4054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:35137: BlockingRecv error: recv got EOF from 127.5.83.65:35137 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 20:16:33 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:33 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:33.213632 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:33.205083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:46757)
0923 20:16:33.205626 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:33.205642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:33.205757 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:33.206102 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:33.206105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:33.206504 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:33.206707 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:33.206716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:33.207801 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:33.207804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:33.208774 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:33.208786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:33.209521 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:33.209525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:33.209654 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:33.210259 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:33.210283 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:33.213392 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":309,"threads_started":1}
W20260923 20:16:33.213989 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:33.205193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:46757 (local address 127.5.83.126:43205)
0923 20:16:33.205383 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:33.205388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:33.205728 (+ 340us) server_negotiation.cc:407] Connection header received
0923 20:16:33.205868 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:33.205872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:33.205974 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:33.206067 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:33.206843 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:33.207673 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:33.208927 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:33.209385 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:33.209499 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:33.213601 (+ 4102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:46757: BlockingRecv error: recv got EOF from 127.5.83.66:46757 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 20:16:33 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:33 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:33.681253 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:33.673198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:50877)
0923 20:16:33.673737 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:33.673753 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:33.673863 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:33.674143 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:33.674146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:33.674542 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:33.674752 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:33.674767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:33.675752 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:33.675756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:33.676767 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:33.676779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:33.677469 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:33.677474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:33.677584 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:33.678154 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:33.678171 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:33.681019 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":297,"threads_started":1}
W20260923 20:16:33.681464 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:33.673281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:50877 (local address 127.5.83.126:43205)
0923 20:16:33.673439 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:33.673443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:33.673824 (+ 381us) server_negotiation.cc:407] Connection header received
0923 20:16:33.673953 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:33.673957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:33.674019 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:33.674104 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:33.674899 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:33.675633 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:33.676895 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:33.677311 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:33.677404 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:33.681216 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:50877: BlockingRecv error: recv got EOF from 127.5.83.67:50877 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 20:16:33 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:33 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:33.744160 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:33.735341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:32801)
0923 20:16:33.735919 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:33.735943 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:33.736094 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:33.736414 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:33.736418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:33.736845 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:33.737008 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:33.737016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:33.738005 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:33.738009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:33.738981 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:33.738994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:33.739694 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:33.739698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:33.739834 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:33.740433 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:33.740452 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:33.743866 (+ 3414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":336,"threads_started":1}
W20260923 20:16:33.744396 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:33.735418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:32801 (local address 127.5.83.126:43205)
0923 20:16:33.735607 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:33.735611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:33.736017 (+ 406us) server_negotiation.cc:407] Connection header received
0923 20:16:33.736197 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:33.736200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:33.736259 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:33.736338 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:33.737134 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:33.737858 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:33.739108 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:33.739568 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:33.739661 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:33.744127 (+ 4466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:32801: BlockingRecv error: recv got EOF from 127.5.83.65:32801 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 20:16:34 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:34 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:34.223382 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:34.214568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:39991)
0923 20:16:34.215108 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:34.215124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:34.215236 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:34.215548 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:34.215551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:34.215933 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:34.216087 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:34.216096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:34.217316 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:34.217320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:34.218275 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:34.218287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:34.219019 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:34.219023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:34.219154 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:34.219812 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:34.219830 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:34.223056 (+ 3226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":297,"threads_started":1}
W20260923 20:16:34.223546 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:34.214609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:39991 (local address 127.5.83.126:43205)
0923 20:16:34.214813 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:34.214818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:34.215225 (+ 407us) server_negotiation.cc:407] Connection header received
0923 20:16:34.215341 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:34.215345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:34.215409 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:34.215502 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:34.216249 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:34.217161 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:34.218424 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:34.218882 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:34.218998 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:34.223290 (+ 4292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:39991: BlockingRecv error: recv got EOF from 127.5.83.66:39991 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 20:16:34 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:34 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:34.690866 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:34.682307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:41239)
0923 20:16:34.682776 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:34.682793 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:34.682908 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:34.683273 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:34.683278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:34.683720 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:34.683916 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:34.683926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:34.685065 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:34.685069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:34.686019 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:34.686030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:34.686739 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:34.686743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:34.686924 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:34.687564 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:34.687583 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:34.690622 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":273,"threads_started":1}
W20260923 20:16:34.691063 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:34.682371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:41239 (local address 127.5.83.126:43205)
0923 20:16:34.682604 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:34.682612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:34.682871 (+ 259us) server_negotiation.cc:407] Connection header received
0923 20:16:34.683016 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:34.683019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:34.683081 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:34.683166 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:34.684075 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:34.684925 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:34.686161 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:34.686606 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:34.686700 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:34.690826 (+ 4126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:41239: BlockingRecv error: recv got EOF from 127.5.83.67:41239 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 23 20:16:34 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:34 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:34.754083 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:34.745173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:38499)
0923 20:16:34.745667 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:34.745684 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:34.745800 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:34.746194 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:34.746198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:34.746606 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:34.746764 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:34.746774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:34.747921 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:34.747925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:34.748962 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:34.748974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:34.749698 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:34.749703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:34.749843 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:34.750485 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:34.750503 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:34.753715 (+ 3212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":279,"threads_started":1}
W20260923 20:16:34.754294 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:34.745261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:38499 (local address 127.5.83.126:43205)
0923 20:16:34.745545 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:34.745552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:34.745767 (+ 215us) server_negotiation.cc:407] Connection header received
0923 20:16:34.745899 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:34.745903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:34.745964 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:34.746087 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:34.746896 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:34.747753 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:34.749095 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:34.749549 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:34.749642 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:34.753909 (+ 4267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:38499: BlockingRecv error: recv got EOF from 127.5.83.65:38499 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 23 20:16:35 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:35 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:35.233196 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:35.224498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:48029)
0923 20:16:35.225167 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:35.225185 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:35.225314 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:35.225674 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:35.225681 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:35.226187 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:35.226357 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:35.226369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:35.227701 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:35.227706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:35.228672 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:35.228683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:35.229409 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:35.229415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:35.229576 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:35.230129 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:35.230146 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:35.232856 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":296,"threads_started":1}
W20260923 20:16:35.233268 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:35.224738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:48029 (local address 127.5.83.126:43205)
0923 20:16:35.224992 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:35.224998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:35.225270 (+ 272us) server_negotiation.cc:407] Connection header received
0923 20:16:35.225422 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:35.225426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:35.225489 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:35.225585 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:35.226513 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:35.227551 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:35.228830 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:35.229228 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:35.229361 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:35.233042 (+ 3681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:48029: BlockingRecv error: recv got EOF from 127.5.83.66:48029 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 23 20:16:35 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:35 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:35.702557 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:35.692147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:33757)
0923 20:16:35.692779 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:35.692795 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:35.692922 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:35.693478 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:35.693484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:35.694092 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:35.694346 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:35.694359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:35.695822 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:35.695835 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:35.697210 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:35.697225 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:35.697986 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:35.697991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:35.698180 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:35.698838 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:35.698857 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:35.702136 (+ 3279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":372,"threads_started":1}
W20260923 20:16:35.702668 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:35.692368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:33757 (local address 127.5.83.126:43205)
0923 20:16:35.692633 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:35.692639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:35.692977 (+ 338us) server_negotiation.cc:407] Connection header received
0923 20:16:35.693098 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:35.693103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:35.693204 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:35.693357 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:35.694600 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:35.695652 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:35.697379 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:35.697823 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:35.697973 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:35.702385 (+ 4412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:33757: BlockingRecv error: recv got EOF from 127.5.83.67:33757 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 23 20:16:35 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:35 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:35.765225 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:35.755149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:57853)
0923 20:16:35.755685 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:35.755704 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:35.755829 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:35.756275 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:35.756278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:35.756725 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:35.756911 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:35.756920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:35.758222 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:35.758231 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:35.759794 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:35.759814 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:35.760722 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:35.760727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:35.760873 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:35.761569 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:35.761589 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:35.764847 (+ 3258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":287,"threads_started":1}
W20260923 20:16:35.765399 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:35.755267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:57853 (local address 127.5.83.126:43205)
0923 20:16:35.755686 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:35.755701 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:35.755804 (+ 103us) server_negotiation.cc:407] Connection header received
0923 20:16:35.756046 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:35.756051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:35.756127 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:35.756231 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:35.757097 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:35.758018 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:35.760031 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:35.760490 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:35.760693 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:35.765103 (+ 4410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:57853: BlockingRecv error: recv got EOF from 127.5.83.65:57853 (error 108)
Metrics: {"server-negotiator.queue_time_us":202}
Sep 23 20:16:36 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:36 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:36.243278 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:36.234140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:48385)
0923 20:16:36.234590 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:36.234605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:36.234722 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:36.235092 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:36.235095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:36.235442 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:36.235621 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:36.235630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:36.236787 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:36.236790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:36.237792 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:36.237811 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:36.238864 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:36.238869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:36.239077 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:36.239848 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:36.239866 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:36.242891 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":245,"threads_started":1}
W20260923 20:16:36.243331 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:36.234283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:48385 (local address 127.5.83.126:43205)
0923 20:16:36.234537 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:36.234544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:36.234687 (+ 143us) server_negotiation.cc:407] Connection header received
0923 20:16:36.234872 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:36.234877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:36.234947 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:36.235049 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:36.235798 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:36.236663 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:36.238010 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:36.238644 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:36.238803 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:36.243113 (+ 4310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:48385: BlockingRecv error: recv got EOF from 127.5.83.66:48385 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 23 20:16:36 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:36 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.67@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:36.716395 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:36.703680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.67:50595)
0923 20:16:36.704270 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:36.704285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:36.704460 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:36.704841 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:36.704846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:36.705235 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:36.705405 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:36.705413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:36.706704 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:36.706711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:36.707825 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:36.707837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:36.708609 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:36.708615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:36.708796 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:36.713029 (+ 4233us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:36.713050 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:36.716065 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":306,"threads_started":1}
W20260923 20:16:36.716625 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:36.703817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:50595 (local address 127.5.83.126:43205)
0923 20:16:36.704105 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:36.704112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:36.704382 (+ 270us) server_negotiation.cc:407] Connection header received
0923 20:16:36.704632 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:36.704641 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:36.704701 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:36.704810 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:36.705565 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:36.706566 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:36.707985 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:36.708423 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:36.708599 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:36.716335 (+ 7736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.67:50595: BlockingRecv error: recv got EOF from 127.5.83.67:50595 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Sep 23 20:16:36 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:36 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.65@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:36.776026 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:36.766473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.65:37539)
0923 20:16:36.767296 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:36.767319 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:36.767475 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:36.767896 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:36.767902 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:36.768611 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:36.768908 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:36.768926 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:36.769959 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:36.769963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:36.770992 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:36.771004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:36.771703 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:36.771708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:36.771840 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:36.772573 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:36.772593 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:36.775631 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":524,"threads_started":1}
W20260923 20:16:36.776093 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:36.766734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:37539 (local address 127.5.83.126:43205)
0923 20:16:36.767006 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:36.767015 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:36.767404 (+ 389us) server_negotiation.cc:407] Connection header received
0923 20:16:36.767603 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:36.767608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:36.767688 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:36.767781 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:36.769078 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:36.769829 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:36.771130 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:36.771555 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:36.771649 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:36.775856 (+ 4207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.65:37539: BlockingRecv error: recv got EOF from 127.5.83.65:37539 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 23 20:16:37 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 20:16:37 dist-test-slave-vl24 krb5kdc[13703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.83.66@KRBTEST.COM for oryx/127.5.83.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 20:16:37.252983 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:37.244594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:43205 (local address 127.5.83.66:35891)
0923 20:16:37.245144 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:37.245161 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:37.245267 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:37.245559 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:37.245562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:37.246019 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:37.246208 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:37.246217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:37.247281 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:37.247285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:37.248509 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:37.248548 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:37.249228 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:37.249233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:37.249353 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:37.249921 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:37.249940 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:37.252733 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.83.126:43205: Server oryx/127.5.83.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":285,"threads_started":1}
W20260923 20:16:37.253223 14440 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:37.244646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:35891 (local address 127.5.83.126:43205)
0923 20:16:37.244828 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:37.244832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:37.245231 (+ 399us) server_negotiation.cc:407] Connection header received
0923 20:16:37.245357 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:37.245360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:37.245417 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:37.245494 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:37.246332 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:37.247140 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:37.248656 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:37.249102 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:37.249194 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:37.252972 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.83.66:35891: BlockingRecv error: recv got EOF from 127.5.83.66:35891 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 20:16:37.533627 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 14291
I20260923 20:16:37.545925 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:37.534651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.5.83.65:32997)
0923 20:16:37.535226 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:37.535233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:37.535258 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:16:37.535336 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:37.535343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:37.535568 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:37.535720 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:37.537284 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:37.538315 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:37.539587 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:37.540105 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:37.540197 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:37.542858 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:37.542866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:37.542879 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:37.542910 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:37.545387 (+ 2477us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.83.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 20:16:37.545657 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35332: Request ticket server kudu/127.5.83.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":292,"threads_started":1}
W20260923 20:16:37.546041 14751 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35332: Request ticket server kudu/127.5.83.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 20:16:37.546196 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 14291: Not authorized: Client connection negotiation failed: client connection to 127.5.83.65:32997: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.83.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 20:16:37.546451 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 14291
I20260923 20:16:37.564785 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 14442
I20260923 20:16:37.578207 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:37.565855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.5.83.66:41605)
0923 20:16:37.566497 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:37.566504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:37.566545 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:16:37.566605 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:37.566612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:37.566829 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:37.566991 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:37.568418 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:37.569595 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:37.570875 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:37.571440 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:37.571545 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:37.574461 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:37.574472 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:37.574486 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:37.574530 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:37.577617 (+ 3087us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.83.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 20:16:37.577949 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58458: Request ticket server kudu/127.5.83.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":302,"threads_started":1}
W20260923 20:16:37.578341 14752 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58458: Request ticket server kudu/127.5.83.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 20:16:37.578503 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 14442: Not authorized: Client connection negotiation failed: client connection to 127.5.83.66:41605: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.83.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 20:16:37.578706 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 14442
I20260923 20:16:37.595865 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 14573
I20260923 20:16:37.609205 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:37.596946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.5.83.67:36453)
0923 20:16:37.597484 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:37.597492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:37.597516 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:16:37.597585 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:37.597592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:37.597827 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:37.597972 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:37.599376 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:37.600460 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:37.601860 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:37.602428 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:37.602528 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:37.605207 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:37.605215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:37.605229 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:37.605282 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:37.608424 (+ 3142us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.83.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 20:16:37.608860 (+ 436us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43732: Request ticket server kudu/127.5.83.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":269,"threads_started":1}
W20260923 20:16:37.609314 14753 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43732: Request ticket server kudu/127.5.83.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 20:16:37.609614 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 14573: Not authorized: Client connection negotiation failed: client connection to 127.5.83.67:36453: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.83.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 20:16:37.609761 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 14573
I20260923 20:16:37.626791 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 14222
I20260923 20:16:37.628151 14284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:37.746035 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 14222
2026-09-23T20:16:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21240 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T20:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:16:37Z Disabled control of system clock
I20260923 20:16:37.827723 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:37415
--webserver_interface=127.5.83.126
--webserver_port=0
--builtin_ntp_servers=127.5.83.84:40335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:37415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 20:16:38.158371 14758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:38.158671 14758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:38.158753 14758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:38.166510 14758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:16:38.166648 14758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:38.166688 14758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:38.166740 14758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:16:38.166786 14758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:16:38.176275 14758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:37415
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:37415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:38.177687 14758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:38.179405 14758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:38.191500 14763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:38.191610 14764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:38.191695 14758 server_base.cc:1034] running on GCE node
W20260923 20:16:38.191500 14766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:38.192735 14758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:38.194108 14758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:38.195304 14758 hybrid_clock.cc:648] HybridClock initialized: now 1790194598195257 us; error 57 us; skew 500 ppm
I20260923 20:16:38.195652 14758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:16:38.198036 14758 webserver.cc:460] Webserver started at http://127.5.83.126:37141/ using document root <none> and password file <none>
I20260923 20:16:38.198735 14758 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:38.198848 14758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:38.199205 14758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:38.202131 14758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "c77ef4d268984374b8ab29a2c0289398"
format_stamp: "Formatted at 2026-09-23 20:16:38 on dist-test-slave-vl24"
server_key: "295855b880413af1ff9a8a4a2fffc0e9"
server_key_iv: "9fb151b1614c37197b72d9e681384839"
server_key_version: "encryptionkey@0"
I20260923 20:16:38.202930 14758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "c77ef4d268984374b8ab29a2c0289398"
format_stamp: "Formatted at 2026-09-23 20:16:38 on dist-test-slave-vl24"
server_key: "295855b880413af1ff9a8a4a2fffc0e9"
server_key_iv: "9fb151b1614c37197b72d9e681384839"
server_key_version: "encryptionkey@0"
I20260923 20:16:38.208115 14758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:16:38.211859 14772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:38.213210 14758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:16:38.213393 14758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "c77ef4d268984374b8ab29a2c0289398"
format_stamp: "Formatted at 2026-09-23 20:16:38 on dist-test-slave-vl24"
server_key: "295855b880413af1ff9a8a4a2fffc0e9"
server_key_iv: "9fb151b1614c37197b72d9e681384839"
server_key_version: "encryptionkey@0"
I20260923 20:16:38.213600 14758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:38.239965 14758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:38.240931 14758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:38.241237 14758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:38.261194 14758 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:37415
I20260923 20:16:38.261235 14823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:37415 every 8 connection(s)
I20260923 20:16:38.262532 14758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:16:38.266223 14824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:38.269342 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 14758
I20260923 20:16:38.269569 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:16:38.270210 5453 external_mini_cluster.cc:1428] Setting key 03727f92aa6b10dbd5b0a06005d5eac3
I20260923 20:16:38.279949 14824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398: Bootstrap starting.
I20260923 20:16:38.283795 14824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:38.285163 14824 log.cc:824] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:38.285651 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:38.272245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.5.83.126:37415)
0923 20:16:38.273444 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:38.273459 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:38.273514 (+ 55us) server_negotiation.cc:407] Connection header received
0923 20:16:38.275370 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:38.275399 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:38.276037 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:38.276473 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:38.277085 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:38.279050 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:38.280499 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:38.281541 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:38.281762 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:38.282803 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:38.282814 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:38.282836 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:38.283524 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:38.283644 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:38.283678 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:38.283798 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:38.284014 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 20:16:38.284362 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":422,"threads_started":1}
I20260923 20:16:38.288270 14824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398: No bootstrap required, opened a new log
I20260923 20:16:38.295686 14824 raft_consensus.cc:348] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77ef4d268984374b8ab29a2c0289398" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37415 } }
I20260923 20:16:38.296059 14824 raft_consensus.cc:374] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:38.296135 14824 raft_consensus.cc:729] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77ef4d268984374b8ab29a2c0289398, State: Initialized, Role: FOLLOWER
I20260923 20:16:38.296783 14824 consensus_queue.cc:260] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77ef4d268984374b8ab29a2c0289398" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37415 } }
I20260923 20:16:38.297024 14824 raft_consensus.cc:388] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:16:38.297163 14824 raft_consensus.cc:482] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:16:38.297353 14824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:38.299638 14824 raft_consensus.cc:504] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77ef4d268984374b8ab29a2c0289398" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37415 } }
I20260923 20:16:38.300091 14824 leader_election.cc:302] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c77ef4d268984374b8ab29a2c0289398; no voters:
I20260923 20:16:38.300709 14824 leader_election.cc:288] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:16:38.300872 14829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:38.301605 14829 raft_consensus.cc:686] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [term 1 LEADER]: Becoming Leader. State: Replica: c77ef4d268984374b8ab29a2c0289398, State: Running, Role: LEADER
I20260923 20:16:38.302150 14829 consensus_queue.cc:237] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77ef4d268984374b8ab29a2c0289398" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37415 } }
I20260923 20:16:38.302883 14824 sys_catalog.cc:564] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:16:38.305431 14830 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c77ef4d268984374b8ab29a2c0289398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77ef4d268984374b8ab29a2c0289398" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37415 } } }
I20260923 20:16:38.305394 14831 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c77ef4d268984374b8ab29a2c0289398. Latest consensus state: current_term: 1 leader_uuid: "c77ef4d268984374b8ab29a2c0289398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77ef4d268984374b8ab29a2c0289398" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37415 } } }
I20260923 20:16:38.305887 14830 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:38.305969 14831 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:38.306686 14835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:16:38.311882 14835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:16:38.321590 14835 catalog_manager.cc:1347] Generated new cluster ID: 3eca35669bc34f349fe5407fc5c6edfd
I20260923 20:16:38.321733 14835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:16:38.331842 14835 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:16:38.333274 14835 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:16:38.342372 14835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c77ef4d268984374b8ab29a2c0289398: Generated new TSK 0
I20260923 20:16:38.343438 14835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:16:38.359016 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37415
--builtin_ntp_servers=127.5.83.84:40335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 20:16:38.691047 14848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:38.691357 14848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:38.691519 14848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:38.698904 14848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:38.699034 14848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:38.699206 14848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:16:38.708175 14848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:38.709587 14848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:38.711663 14848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:38.725097 14853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:38.725248 14856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:38.725274 14854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:38.725788 14848 server_base.cc:1034] running on GCE node
I20260923 20:16:38.726321 14848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:38.727211 14848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:38.728521 14848 hybrid_clock.cc:648] HybridClock initialized: now 1790194598728446 us; error 50 us; skew 500 ppm
I20260923 20:16:38.728961 14848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:16:38.731232 14848 webserver.cc:460] Webserver started at http://127.5.83.65:43593/ using document root <none> and password file <none>
I20260923 20:16:38.731935 14848 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:38.732023 14848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:38.732371 14848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:38.735183 14848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "3cf72f099ac84aff9381c44db8dcd9ef"
format_stamp: "Formatted at 2026-09-23 20:16:38 on dist-test-slave-vl24"
server_key: "ff25b72539986fc7a83621ce0b6fb962"
server_key_iv: "54c425f539b970a7bbfb017eacc96c49"
server_key_version: "encryptionkey@0"
I20260923 20:16:38.735960 14848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "3cf72f099ac84aff9381c44db8dcd9ef"
format_stamp: "Formatted at 2026-09-23 20:16:38 on dist-test-slave-vl24"
server_key: "ff25b72539986fc7a83621ce0b6fb962"
server_key_iv: "54c425f539b970a7bbfb017eacc96c49"
server_key_version: "encryptionkey@0"
I20260923 20:16:38.741051 14848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260923 20:16:38.744625 14862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:38.745954 14848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:16:38.746135 14848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "3cf72f099ac84aff9381c44db8dcd9ef"
format_stamp: "Formatted at 2026-09-23 20:16:38 on dist-test-slave-vl24"
server_key: "ff25b72539986fc7a83621ce0b6fb962"
server_key_iv: "54c425f539b970a7bbfb017eacc96c49"
server_key_version: "encryptionkey@0"
I20260923 20:16:38.746349 14848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:38.783502 14848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:38.784448 14848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:38.784811 14848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:38.785795 14848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:38.787438 14848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:38.787570 14848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:38.787688 14848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:38.787778 14848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:38.821449 14848 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:33477
I20260923 20:16:38.821609 14974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:33477 every 8 connection(s)
I20260923 20:16:38.822739 14848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:16:38.825831 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 14848
I20260923 20:16:38.826057 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:16:38.826946 5453 external_mini_cluster.cc:1428] Setting key d50f9d0f13b245ed821c0be421459348
I20260923 20:16:38.829710 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37415
--builtin_ntp_servers=127.5.83.84:40335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 20:16:38.837001 14977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:38.825601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:47523 (local address 127.5.83.126:37415)
0923 20:16:38.826151 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:38.826157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:38.827651 (+ 1494us) server_negotiation.cc:407] Connection header received
0923 20:16:38.829592 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:38.829597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:38.829683 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:38.829808 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:38.830896 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:38.832077 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:38.833688 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:38.834391 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:38.834539 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:38.835940 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:38.835948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:38.835956 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:38.835987 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:38.836043 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:38.836062 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:38.836161 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:38.836395 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 20:16:38.836507 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":273,"threads_started":1}
I20260923 20:16:38.838074 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:38.826335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:37415 (local address 127.5.83.65:47523)
0923 20:16:38.827397 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:38.827438 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:38.829315 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:38.830091 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:38.830099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:38.830183 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:38.830722 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:38.830738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:38.832250 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:38.832254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:38.833551 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:38.833562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:38.834619 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:38.834624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:38.834831 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:38.835581 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:16:38.835583 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:38.835622 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:16:38.835700 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:38.836163 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:38.836171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:38.836176 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:16:38.836355 (+ 179us) client_negotiation.cc:240] Negotiation successful
0923 20:16:38.836740 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":391,"threads_started":1}
I20260923 20:16:38.839637 14975 heartbeater.cc:347] Connected to a master server at 127.5.83.126:37415
I20260923 20:16:38.839969 14975 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:38.840850 14975 heartbeater.cc:511] Master 127.5.83.126:37415 requested a full tablet report, sending...
I20260923 20:16:38.843230 14789 ts_manager.cc:194] Registered new tserver with Master: 3cf72f099ac84aff9381c44db8dcd9ef (127.5.83.65:33477)
I20260923 20:16:38.844874 14789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.65:47523
W20260923 20:16:39.161433 14979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:39.161777 14979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:39.161938 14979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:39.171782 14979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:39.171950 14979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:39.172134 14979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:16:39.181799 14979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:39.183168 14979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:39.185047 14979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:39.198230 14984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:39.198242 14985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:39.198242 14987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:39.199234 14979 server_base.cc:1034] running on GCE node
I20260923 20:16:39.199823 14979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:39.200809 14979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:39.202036 14979 hybrid_clock.cc:648] HybridClock initialized: now 1790194599201987 us; error 54 us; skew 500 ppm
I20260923 20:16:39.202342 14979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:16:39.204610 14979 webserver.cc:460] Webserver started at http://127.5.83.66:36885/ using document root <none> and password file <none>
I20260923 20:16:39.205344 14979 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:39.205477 14979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:39.205807 14979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:39.208856 14979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "889214149f9545ada19593f657c966a7"
format_stamp: "Formatted at 2026-09-23 20:16:39 on dist-test-slave-vl24"
server_key: "35dabc76cfb1652bea5126a5b89aa359"
server_key_iv: "41d4ccbdc3f1234d5656e7ff4559e9b7"
server_key_version: "encryptionkey@0"
I20260923 20:16:39.209687 14979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "889214149f9545ada19593f657c966a7"
format_stamp: "Formatted at 2026-09-23 20:16:39 on dist-test-slave-vl24"
server_key: "35dabc76cfb1652bea5126a5b89aa359"
server_key_iv: "41d4ccbdc3f1234d5656e7ff4559e9b7"
server_key_version: "encryptionkey@0"
I20260923 20:16:39.214835 14979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:16:39.218513 14993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:39.219826 14979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:16:39.220000 14979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "889214149f9545ada19593f657c966a7"
format_stamp: "Formatted at 2026-09-23 20:16:39 on dist-test-slave-vl24"
server_key: "35dabc76cfb1652bea5126a5b89aa359"
server_key_iv: "41d4ccbdc3f1234d5656e7ff4559e9b7"
server_key_version: "encryptionkey@0"
I20260923 20:16:39.220225 14979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:39.234750 14979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:39.235718 14979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:39.236065 14979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:39.237257 14979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:39.238783 14979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:39.238924 14979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:39.239087 14979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:39.239195 14979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:39.274701 14979 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:34851
I20260923 20:16:39.274739 15105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:34851 every 8 connection(s)
I20260923 20:16:39.276405 14979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:16:39.276909 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 14979
I20260923 20:16:39.277119 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:16:39.277678 5453 external_mini_cluster.cc:1428] Setting key 1ff0965ce59b4f01c07b0c8f92b08973
I20260923 20:16:39.281157 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37415
--builtin_ntp_servers=127.5.83.84:40335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 20:16:39.302927 14977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:39.280760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:46543 (local address 127.5.83.126:37415)
0923 20:16:39.280947 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:39.280954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:39.291025 (+ 10071us) server_negotiation.cc:407] Connection header received
0923 20:16:39.294759 (+ 3734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:39.294764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:39.294838 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:39.294938 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:39.296449 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.297287 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.299868 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.300242 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.300340 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:39.302106 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:39.302111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:39.302114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:39.302134 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:39.302166 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:39.302178 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:39.302252 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:39.302515 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 20:16:39.302605 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 20:16:39.304450 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:39.280175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:37415 (local address 127.5.83.66:46543)
0923 20:16:39.290824 (+ 10649us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:39.290882 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:39.294406 (+ 3524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:39.295223 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:39.295239 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:39.295371 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:39.296251 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:39.296280 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.297468 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.297477 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:39.299712 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:39.299730 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.300485 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.300495 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:39.300860 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:39.301688 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:16:39.301693 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:39.301758 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:16:39.301868 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:39.302278 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:39.302292 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:39.302303 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:16:39.302498 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 20:16:39.302991 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9971,"thread_start_us":10769,"threads_started":1}
I20260923 20:16:39.306021 15106 heartbeater.cc:347] Connected to a master server at 127.5.83.126:37415
I20260923 20:16:39.306341 15106 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:39.307112 15106 heartbeater.cc:511] Master 127.5.83.126:37415 requested a full tablet report, sending...
I20260923 20:16:39.308634 14789 ts_manager.cc:194] Registered new tserver with Master: 889214149f9545ada19593f657c966a7 (127.5.83.66:34851)
I20260923 20:16:39.309466 14789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.66:46543
W20260923 20:16:39.622908 15109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:39.623294 15109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:39.623435 15109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:39.630899 15109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:39.631059 15109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:39.631271 15109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:16:39.640707 15109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:39.642032 15109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:39.643797 15109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:39.657616 15114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:39.657886 15117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:39.657665 15115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:39.658746 15109 server_base.cc:1034] running on GCE node
I20260923 20:16:39.659426 15109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:39.660399 15109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:39.661636 15109 hybrid_clock.cc:648] HybridClock initialized: now 1790194599661610 us; error 64 us; skew 500 ppm
I20260923 20:16:39.661964 15109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:16:39.664320 15109 webserver.cc:460] Webserver started at http://127.5.83.67:33733/ using document root <none> and password file <none>
I20260923 20:16:39.665081 15109 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:39.665213 15109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:39.665555 15109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:39.668393 15109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "194595c2d76d4cdc9742a16058f29c6c"
format_stamp: "Formatted at 2026-09-23 20:16:39 on dist-test-slave-vl24"
server_key: "a38a12b2fc79f7d08d2802fcab302f58"
server_key_iv: "dd14d8179546052a490b913eb54b99b3"
server_key_version: "encryptionkey@0"
I20260923 20:16:39.669258 15109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "194595c2d76d4cdc9742a16058f29c6c"
format_stamp: "Formatted at 2026-09-23 20:16:39 on dist-test-slave-vl24"
server_key: "a38a12b2fc79f7d08d2802fcab302f58"
server_key_iv: "dd14d8179546052a490b913eb54b99b3"
server_key_version: "encryptionkey@0"
I20260923 20:16:39.674237 15109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:16:39.677489 15123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:39.678781 15109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:16:39.678954 15109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "194595c2d76d4cdc9742a16058f29c6c"
format_stamp: "Formatted at 2026-09-23 20:16:39 on dist-test-slave-vl24"
server_key: "a38a12b2fc79f7d08d2802fcab302f58"
server_key_iv: "dd14d8179546052a490b913eb54b99b3"
server_key_version: "encryptionkey@0"
I20260923 20:16:39.679190 15109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:39.692619 15109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:39.693509 15109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:39.693811 15109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:39.694749 15109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:39.696223 15109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:39.696347 15109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:39.696483 15109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:39.696591 15109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:39.730844 15109 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:44409
I20260923 20:16:39.730885 15235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:44409 every 8 connection(s)
I20260923 20:16:39.732312 15109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:16:39.733306 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 15109
I20260923 20:16:39.733511 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:16:39.734082 5453 external_mini_cluster.cc:1428] Setting key 89a03898d653ddfaa70228d6811a0572
I20260923 20:16:39.745152 14977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:39.735394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:59041 (local address 127.5.83.126:37415)
0923 20:16:39.735617 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:39.735623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:39.737270 (+ 1647us) server_negotiation.cc:407] Connection header received
0923 20:16:39.738992 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:39.738997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:39.739086 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:39.739190 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:39.740189 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.740989 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.742465 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.742862 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.742990 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:39.744337 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:39.744342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:39.744345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:39.744363 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:39.744398 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:39.744409 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:39.744505 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:39.744744 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 20:16:39.744836 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:16:39.746189 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:39.736028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:37415 (local address 127.5.83.67:59041)
0923 20:16:39.737124 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:39.737168 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:39.738723 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:39.739415 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:39.739423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:39.739503 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:39.740039 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:39.740055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.741126 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.741138 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:39.742317 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:39.742328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.743068 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.743074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:39.743301 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:39.744031 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:16:39.744034 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:39.744074 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:16:39.744143 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:39.744501 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:39.744512 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:39.744522 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:16:39.744735 (+ 213us) client_negotiation.cc:240] Negotiation successful
0923 20:16:39.745101 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":358,"threads_started":1}
I20260923 20:16:39.747681 15236 heartbeater.cc:347] Connected to a master server at 127.5.83.126:37415
I20260923 20:16:39.747972 15236 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:39.749066 15236 heartbeater.cc:511] Master 127.5.83.126:37415 requested a full tablet report, sending...
I20260923 20:16:39.750662 14789 ts_manager.cc:194] Registered new tserver with Master: 194595c2d76d4cdc9742a16058f29c6c (127.5.83.67:44409)
I20260923 20:16:39.751478 14789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.67:59041
I20260923 20:16:39.762590 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 20:16:39.768666 14977 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:39.767017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.5.83.126:37415)
0923 20:16:39.767299 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:39.767308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:39.767673 (+ 365us) server_negotiation.cc:407] Connection header received
0923 20:16:39.767811 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:39.767814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:39.767890 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:39.767983 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:39.768369 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40270: BlockingRecv error: recv got EOF from 127.0.0.1:40270 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:16:39.769039 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 14848
I20260923 20:16:39.776697 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:39.769988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.5.83.65:33477)
0923 20:16:39.770490 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:39.770496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:39.770521 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:16:39.770579 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:39.770585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:39.770788 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:39.770913 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:39.771274 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.772303 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.773672 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.774203 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.774297 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:39.775311 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:39.775319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:39.775332 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:39.775904 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:39.775959 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:39.775977 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:39.776073 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:39.776212 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 20:16:39.776335 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":257,"threads_started":1}
I20260923 20:16:39.777913 14970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:39.859346 14975 heartbeater.cc:503] Master 127.5.83.126:37415 was elected leader, sending a full tablet report...
I20260923 20:16:39.861086 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 14848
I20260923 20:16:39.879014 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 14979
I20260923 20:16:39.887028 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:39.879894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.5.83.66:34851)
0923 20:16:39.880406 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:39.880413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:39.880440 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:16:39.880505 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:39.880511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:39.880792 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:39.880976 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:39.881353 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.882400 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.883772 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.884352 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.884492 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:39.885602 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:39.885609 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:39.885621 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:39.886180 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:39.886223 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:39.886246 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:39.886376 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:39.886539 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 20:16:39.886679 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":252,"threads_started":1}
I20260923 20:16:39.888239 15101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:39.969774 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 14979
I20260923 20:16:39.986387 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 15109
I20260923 20:16:39.994975 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:39.987351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54284 (local address 127.5.83.67:44409)
0923 20:16:39.987986 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:39.987992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:39.988023 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:16:39.988102 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:39.988109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:39.988319 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:39.988443 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:39.989021 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.990063 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.991643 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:39.992245 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:39.992372 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:39.993524 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:39.993534 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:39.993551 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:39.994167 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:39.994214 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:39.994240 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:39.994365 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:39.994440 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 20:16:39.994596 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":346,"threads_started":1}
I20260923 20:16:39.996289 15231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:40.079911 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 15109
I20260923 20:16:40.096268 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 14758
I20260923 20:16:40.097560 14819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:40.200488 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 14758
2026-09-23T20:16:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2448 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T20:16:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:16:40Z Disabled control of system clock
I20260923 20:16:40.277419 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:46781
--webserver_interface=127.5.83.126
--webserver_port=0
--builtin_ntp_servers=127.5.83.84:40303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:46781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 20:16:40.612711 15255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:40.612998 15255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:40.613171 15255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:40.621701 15255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:16:40.621822 15255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:40.621888 15255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:40.621966 15255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:16:40.622035 15255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:16:40.631417 15255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:46781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:46781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:40.632807 15255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:40.634591 15255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:40.646533 15260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:40.646584 15263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:40.646636 15261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:40.647791 15255 server_base.cc:1034] running on GCE node
I20260923 20:16:40.648664 15255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:40.650163 15255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:40.651368 15255 hybrid_clock.cc:648] HybridClock initialized: now 1790194600651332 us; error 39 us; skew 500 ppm
I20260923 20:16:40.651714 15255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:16:40.654035 15255 webserver.cc:460] Webserver started at http://127.5.83.126:32823/ using document root <none> and password file <none>
I20260923 20:16:40.654748 15255 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:40.654860 15255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:40.655187 15255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:40.659639 15255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "033e19d804ad4e448a89f14846c6fcf8"
format_stamp: "Formatted at 2026-09-23 20:16:40 on dist-test-slave-vl24"
server_key: "a43a2ce5f598570056f39b0d70557044"
server_key_iv: "5467ac8b1d881aeea099ad747b9a88ea"
server_key_version: "encryptionkey@0"
I20260923 20:16:40.660406 15255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "033e19d804ad4e448a89f14846c6fcf8"
format_stamp: "Formatted at 2026-09-23 20:16:40 on dist-test-slave-vl24"
server_key: "a43a2ce5f598570056f39b0d70557044"
server_key_iv: "5467ac8b1d881aeea099ad747b9a88ea"
server_key_version: "encryptionkey@0"
I20260923 20:16:40.665551 15255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260923 20:16:40.669101 15269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:40.670543 15255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:16:40.670732 15255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "033e19d804ad4e448a89f14846c6fcf8"
format_stamp: "Formatted at 2026-09-23 20:16:40 on dist-test-slave-vl24"
server_key: "a43a2ce5f598570056f39b0d70557044"
server_key_iv: "5467ac8b1d881aeea099ad747b9a88ea"
server_key_version: "encryptionkey@0"
I20260923 20:16:40.670954 15255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:40.676770 15255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:40.677543 15255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:40.677834 15255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:40.696745 15255 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:46781
I20260923 20:16:40.696787 15320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:46781 every 8 connection(s)
I20260923 20:16:40.698164 15255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:16:40.700584 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 15255
I20260923 20:16:40.700804 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:16:40.701727 5453 external_mini_cluster.cc:1428] Setting key 8e1006cfdfb27d2a7cd9b1275a7f5a6e
I20260923 20:16:40.702447 15321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:40.712028 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:40.704134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.5.83.126:46781)
0923 20:16:40.705572 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:40.705587 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:40.705643 (+ 56us) server_negotiation.cc:407] Connection header received
0923 20:16:40.707769 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:40.707799 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:40.708382 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:40.708886 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:40.709964 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:40.709973 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:40.709990 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:40.710007 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:40.710099 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:40.710121 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:40.710352 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:40.710510 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 20:16:40.710802 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":649,"threads_started":1}
I20260923 20:16:40.717105 15321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8: Bootstrap starting.
I20260923 20:16:40.720134 15321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:40.721232 15321 log.cc:824] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:40.724212 15321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8: No bootstrap required, opened a new log
I20260923 20:16:40.731859 15321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033e19d804ad4e448a89f14846c6fcf8" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46781 } }
I20260923 20:16:40.732203 15321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:40.732275 15321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 033e19d804ad4e448a89f14846c6fcf8, State: Initialized, Role: FOLLOWER
I20260923 20:16:40.732833 15321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033e19d804ad4e448a89f14846c6fcf8" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46781 } }
I20260923 20:16:40.733042 15321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:16:40.733139 15321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:16:40.733258 15321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:40.735558 15321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033e19d804ad4e448a89f14846c6fcf8" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46781 } }
I20260923 20:16:40.736117 15321 leader_election.cc:302] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 033e19d804ad4e448a89f14846c6fcf8; no voters:
I20260923 20:16:40.736833 15321 leader_election.cc:288] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:16:40.736979 15326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:40.737749 15326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [term 1 LEADER]: Becoming Leader. State: Replica: 033e19d804ad4e448a89f14846c6fcf8, State: Running, Role: LEADER
I20260923 20:16:40.738288 15326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033e19d804ad4e448a89f14846c6fcf8" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46781 } }
I20260923 20:16:40.738988 15321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:16:40.741236 15327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "033e19d804ad4e448a89f14846c6fcf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033e19d804ad4e448a89f14846c6fcf8" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46781 } } }
I20260923 20:16:40.741688 15327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:40.742311 15328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 033e19d804ad4e448a89f14846c6fcf8. Latest consensus state: current_term: 1 leader_uuid: "033e19d804ad4e448a89f14846c6fcf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "033e19d804ad4e448a89f14846c6fcf8" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46781 } } }
I20260923 20:16:40.742693 15328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:40.744692 15335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:16:40.748767 15335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:16:40.757625 15335 catalog_manager.cc:1347] Generated new cluster ID: dabe1f1ab5b24fe592e009848b41b355
I20260923 20:16:40.757747 15335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:16:40.778782 15335 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:16:40.779273 15335 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:16:40.791388 15335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 033e19d804ad4e448a89f14846c6fcf8: Generated new TSK 0
I20260923 20:16:40.792457 15335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:16:40.811372 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46781
--builtin_ntp_servers=127.5.83.84:40303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 20:16:41.154436 15345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:41.154727 15345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:41.154883 15345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:41.162447 15345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:41.162606 15345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:41.162803 15345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:16:41.174584 15345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:41.176041 15345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:41.177977 15345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:41.191249 15351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:41.191228 15353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:41.191306 15350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:41.192122 15345 server_base.cc:1034] running on GCE node
I20260923 20:16:41.192798 15345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:41.193686 15345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:41.194880 15345 hybrid_clock.cc:648] HybridClock initialized: now 1790194601194805 us; error 63 us; skew 500 ppm
I20260923 20:16:41.195199 15345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:16:41.197628 15345 webserver.cc:460] Webserver started at http://127.5.83.65:39709/ using document root <none> and password file <none>
I20260923 20:16:41.198311 15345 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:41.198421 15345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:41.198777 15345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:41.201768 15345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "dc4181f273e640d9bb8ddeae00aac097"
format_stamp: "Formatted at 2026-09-23 20:16:41 on dist-test-slave-vl24"
server_key: "06860a2abd1c35302f5b0ca6e30d58d6"
server_key_iv: "df95d835f4d66e6bcaee4b457ce1fe88"
server_key_version: "encryptionkey@0"
I20260923 20:16:41.202555 15345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "dc4181f273e640d9bb8ddeae00aac097"
format_stamp: "Formatted at 2026-09-23 20:16:41 on dist-test-slave-vl24"
server_key: "06860a2abd1c35302f5b0ca6e30d58d6"
server_key_iv: "df95d835f4d66e6bcaee4b457ce1fe88"
server_key_version: "encryptionkey@0"
I20260923 20:16:41.207810 15345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260923 20:16:41.211566 15359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:41.213156 15345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:16:41.213342 15345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "dc4181f273e640d9bb8ddeae00aac097"
format_stamp: "Formatted at 2026-09-23 20:16:41 on dist-test-slave-vl24"
server_key: "06860a2abd1c35302f5b0ca6e30d58d6"
server_key_iv: "df95d835f4d66e6bcaee4b457ce1fe88"
server_key_version: "encryptionkey@0"
I20260923 20:16:41.213586 15345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:41.219928 15345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:41.220783 15345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:41.221056 15345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:41.222000 15345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:41.223423 15345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:41.223544 15345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:41.223685 15345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:41.223769 15345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:41.258586 15345 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:40585
I20260923 20:16:41.258637 15471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:40585 every 8 connection(s)
I20260923 20:16:41.259883 15345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:16:41.266382 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 15345
I20260923 20:16:41.266575 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:16:41.267220 5453 external_mini_cluster.cc:1428] Setting key 2cac200097361f1a0571268cc92772fc
I20260923 20:16:41.268211 15474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:41.262515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:50499 (local address 127.5.83.126:46781)
0923 20:16:41.262950 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:41.262956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:41.264105 (+ 1149us) server_negotiation.cc:407] Connection header received
0923 20:16:41.265840 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:41.265845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:41.265938 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:41.266057 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:41.267454 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:41.267459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:41.267464 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:41.267485 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:41.267538 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:41.267556 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:41.267689 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:41.267868 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 20:16:41.267924 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":213,"threads_started":1}
I20260923 20:16:41.269265 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:41.263083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46781 (local address 127.5.83.65:50499)
0923 20:16:41.263952 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:41.263995 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:41.265576 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:41.266278 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:41.266286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:41.266369 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:41.267094 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:16:41.267096 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:41.267135 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:16:41.267214 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:41.267670 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:41.267678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:41.267684 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:16:41.267927 (+ 243us) client_negotiation.cc:240] Negotiation successful
0923 20:16:41.268192 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":272,"threads_started":1}
I20260923 20:16:41.270300 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46781
--builtin_ntp_servers=127.5.83.84:40303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 20:16:41.270875 15472 heartbeater.cc:347] Connected to a master server at 127.5.83.126:46781
I20260923 20:16:41.271194 15472 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:41.271878 15472 heartbeater.cc:511] Master 127.5.83.126:46781 requested a full tablet report, sending...
I20260923 20:16:41.274121 15286 ts_manager.cc:194] Registered new tserver with Master: dc4181f273e640d9bb8ddeae00aac097 (127.5.83.65:40585)
W20260923 20:16:41.598280 15476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:41.598584 15476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:41.598726 15476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:41.606122 15476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:41.606247 15476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:41.606446 15476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:16:41.616137 15476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:41.617591 15476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:41.619457 15476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:41.633627 15481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:41.633721 15484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:41.633685 15482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:41.634785 15476 server_base.cc:1034] running on GCE node
I20260923 20:16:41.635428 15476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:41.636385 15476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:41.637601 15476 hybrid_clock.cc:648] HybridClock initialized: now 1790194601637534 us; error 69 us; skew 500 ppm
I20260923 20:16:41.637964 15476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:16:41.640599 15476 webserver.cc:460] Webserver started at http://127.5.83.66:44307/ using document root <none> and password file <none>
I20260923 20:16:41.641362 15476 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:41.641474 15476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:41.641880 15476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:41.645113 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "a1e8d7da00a84e809beea399d0ecb1fc"
format_stamp: "Formatted at 2026-09-23 20:16:41 on dist-test-slave-vl24"
server_key: "462f96800bb46a837b6e8567e9bcce73"
server_key_iv: "39b012d8a69abeafba1f48f9ece38c48"
server_key_version: "encryptionkey@0"
I20260923 20:16:41.645931 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "a1e8d7da00a84e809beea399d0ecb1fc"
format_stamp: "Formatted at 2026-09-23 20:16:41 on dist-test-slave-vl24"
server_key: "462f96800bb46a837b6e8567e9bcce73"
server_key_iv: "39b012d8a69abeafba1f48f9ece38c48"
server_key_version: "encryptionkey@0"
I20260923 20:16:41.651433 15476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:16:41.655270 15490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:41.656957 15476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:16:41.657166 15476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "a1e8d7da00a84e809beea399d0ecb1fc"
format_stamp: "Formatted at 2026-09-23 20:16:41 on dist-test-slave-vl24"
server_key: "462f96800bb46a837b6e8567e9bcce73"
server_key_iv: "39b012d8a69abeafba1f48f9ece38c48"
server_key_version: "encryptionkey@0"
I20260923 20:16:41.657392 15476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:41.663584 15476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:41.664434 15476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:41.664774 15476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:41.665733 15476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:41.667168 15476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:41.667279 15476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:41.667413 15476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:41.667507 15476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:41.702740 15476 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:42507
I20260923 20:16:41.702783 15602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:42507 every 8 connection(s)
I20260923 20:16:41.704043 15476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:16:41.705760 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 15476
I20260923 20:16:41.706137 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:16:41.706753 5453 external_mini_cluster.cc:1428] Setting key 6c05bcaa219e40a95144af4dc396e459
I20260923 20:16:41.709913 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46781
--builtin_ntp_servers=127.5.83.84:40303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 20:16:41.714039 15474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:41.706793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:47915 (local address 127.5.83.126:46781)
0923 20:16:41.707462 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:41.707470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:41.708975 (+ 1505us) server_negotiation.cc:407] Connection header received
0923 20:16:41.711537 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:41.711543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:41.711622 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:41.711726 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:41.713253 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:41.713259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:41.713264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:41.713284 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:41.713323 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:41.713339 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:41.713478 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:41.713702 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 20:16:41.713760 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531}
I20260923 20:16:41.715217 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:41.707453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46781 (local address 127.5.83.66:47915)
0923 20:16:41.708813 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:41.708858 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:41.711241 (+ 2383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:41.711977 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:41.711990 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:41.712122 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:41.712907 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:16:41.712913 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:41.712961 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:16:41.713079 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:41.713467 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:41.713478 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:41.713488 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:16:41.713764 (+ 276us) client_negotiation.cc:240] Negotiation successful
0923 20:16:41.714027 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":398,"threads_started":1}
I20260923 20:16:41.716730 15603 heartbeater.cc:347] Connected to a master server at 127.5.83.126:46781
I20260923 20:16:41.717056 15603 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:41.717720 15603 heartbeater.cc:511] Master 127.5.83.126:46781 requested a full tablet report, sending...
I20260923 20:16:41.719357 15286 ts_manager.cc:194] Registered new tserver with Master: a1e8d7da00a84e809beea399d0ecb1fc (127.5.83.66:42507)
W20260923 20:16:42.041363 15606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:42.041738 15606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:42.041899 15606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:42.049290 15606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:42.049417 15606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:42.049592 15606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:16:42.058878 15606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:42.060176 15606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:42.062071 15606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:42.075559 15612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:42.075515 15611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:42.076186 15614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:42.076432 15606 server_base.cc:1034] running on GCE node
I20260923 20:16:42.077028 15606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:42.077939 15606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:42.079152 15606 hybrid_clock.cc:648] HybridClock initialized: now 1790194602079114 us; error 45 us; skew 500 ppm
I20260923 20:16:42.079452 15606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:16:42.085466 15606 webserver.cc:460] Webserver started at http://127.5.83.67:32853/ using document root <none> and password file <none>
I20260923 20:16:42.086194 15606 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:42.086325 15606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:42.086650 15606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:42.089574 15606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "211703a0a3364307ae29097febf8f824"
format_stamp: "Formatted at 2026-09-23 20:16:42 on dist-test-slave-vl24"
server_key: "9bfdf2e2e19db1270dba9f7298f05f51"
server_key_iv: "c14f4bc2e690cf2023ec61447a15eb8c"
server_key_version: "encryptionkey@0"
I20260923 20:16:42.090353 15606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "211703a0a3364307ae29097febf8f824"
format_stamp: "Formatted at 2026-09-23 20:16:42 on dist-test-slave-vl24"
server_key: "9bfdf2e2e19db1270dba9f7298f05f51"
server_key_iv: "c14f4bc2e690cf2023ec61447a15eb8c"
server_key_version: "encryptionkey@0"
I20260923 20:16:42.095399 15606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:16:42.098838 15620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:42.100140 15606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:16:42.100319 15606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "211703a0a3364307ae29097febf8f824"
format_stamp: "Formatted at 2026-09-23 20:16:42 on dist-test-slave-vl24"
server_key: "9bfdf2e2e19db1270dba9f7298f05f51"
server_key_iv: "c14f4bc2e690cf2023ec61447a15eb8c"
server_key_version: "encryptionkey@0"
I20260923 20:16:42.100564 15606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:42.106356 15606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:42.107153 15606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:42.107424 15606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:42.108325 15606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:42.109722 15606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:42.109812 15606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:42.109886 15606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:42.109941 15606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:42.143767 15606 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:36725
I20260923 20:16:42.143841 15732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:36725 every 8 connection(s)
I20260923 20:16:42.145197 15606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:16:42.146589 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 15606
I20260923 20:16:42.146759 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:16:42.147222 5453 external_mini_cluster.cc:1428] Setting key b1d7d8c8cbb79b0d2790b558b2da757b
I20260923 20:16:42.153422 15474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:42.147716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:54431 (local address 127.5.83.126:46781)
0923 20:16:42.147922 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:42.147930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:42.149694 (+ 1764us) server_negotiation.cc:407] Connection header received
0923 20:16:42.151326 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:42.151330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:42.151473 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:42.151587 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:42.152851 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:42.152855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:42.152858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:42.152873 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:42.152906 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:42.152917 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:42.153006 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:42.153157 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 20:16:42.153204 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 20:16:42.154457 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:42.148397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46781 (local address 127.5.83.67:54431)
0923 20:16:42.149533 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:42.149577 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:42.151081 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:42.151753 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:42.151761 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:42.151850 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:42.152479 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:16:42.152482 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:42.152585 (+ 103us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:16:42.152692 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:42.152984 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:42.152989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:42.152995 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:16:42.153227 (+ 232us) client_negotiation.cc:240] Negotiation successful
0923 20:16:42.153447 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":374,"threads_started":1}
I20260923 20:16:42.155786 15733 heartbeater.cc:347] Connected to a master server at 127.5.83.126:46781
I20260923 20:16:42.156098 15733 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:42.156755 15733 heartbeater.cc:511] Master 127.5.83.126:46781 requested a full tablet report, sending...
I20260923 20:16:42.158092 15286 ts_manager.cc:194] Registered new tserver with Master: 211703a0a3364307ae29097febf8f824 (127.5.83.67:36725)
I20260923 20:16:42.162451 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 20:16:42.168695 15474 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:16:42.166947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.5.83.126:46781)
0923 20:16:42.167114 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:42.167119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:42.167633 (+ 514us) server_negotiation.cc:407] Connection header received
0923 20:16:42.167826 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:42.167831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:42.167893 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:42.167996 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:42.168374 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51034: BlockingRecv error: recv got EOF from 127.0.0.1:51034 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:16:42.169819 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 15345
I20260923 20:16:42.176707 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:42.170810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.5.83.65:40585)
0923 20:16:42.171356 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:42.171363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:42.174188 (+ 2825us) server_negotiation.cc:407] Connection header received
0923 20:16:42.174428 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:42.174436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:42.174682 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:42.174856 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:42.175927 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:42.175933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:42.175944 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:42.175963 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:42.176002 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:42.176015 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:42.176163 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:42.176242 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 20:16:42.176343 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":276,"threads_started":1}
I20260923 20:16:42.177981 15467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:42.260521 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 15345
I20260923 20:16:42.277083 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 15476
I20260923 20:16:42.280786 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:42.278011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.5.83.66:42507)
0923 20:16:42.278525 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:42.278539 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:42.278564 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:16:42.278624 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:42.278630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:42.278843 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:42.279028 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:42.280064 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:42.280071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:42.280082 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:42.280094 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:42.280131 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:42.280144 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:42.280286 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:42.280389 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 20:16:42.280485 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":264,"threads_started":1}
I20260923 20:16:42.281968 15598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:42.364035 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 15476
I20260923 20:16:42.380380 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 15606
I20260923 20:16:42.384490 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:42.381407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.5.83.67:36725)
0923 20:16:42.382088 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:42.382095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:42.382126 (+ 31us) server_negotiation.cc:407] Connection header received
0923 20:16:42.382185 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:42.382191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:42.382415 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:42.382601 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:42.383738 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:42.383744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:42.383755 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:16:42.383771 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:42.383809 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:16:42.383822 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:42.384022 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:42.384096 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 20:16:42.384196 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":350,"threads_started":1}
I20260923 20:16:42.385768 15728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:42.468189 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 15606
I20260923 20:16:42.484398 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 15255
I20260923 20:16:42.485631 15316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:42.587698 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 15255
2026-09-23T20:16:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2387 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:16:42 dist-test-slave-vl24 krb5kdc[15751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:16:42 dist-test-slave-vl24 krb5kdc[15751](info): set up 2 sockets
Sep 23 20:16:42 dist-test-slave-vl24 krb5kdc[15751](info): commencing operation
krb5kdc: starting...
W20260923 20:16:43.722748 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:16:43 dist-test-slave-vl24 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:16:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:16:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:16:43.977561 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:35181
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:35181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:44.329531 15764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:44.329829 15764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:44.329989 15764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:44.337364 15764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:16:44.337476 15764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:44.337586 15764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:44.337688 15764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:16:44.337773 15764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:16:44.347749 15764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:35181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:35181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:44.349263 15764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:44.350560 15764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:44.364507 15769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:44.364495 15772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:44.364459 15770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:44.365305 15764 server_base.cc:1034] running on GCE node
I20260923 20:16:44.366312 15764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:44.367951 15764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:44.369223 15764 hybrid_clock.cc:648] HybridClock initialized: now 1790194604369118 us; error 79 us; skew 500 ppm
I20260923 20:16:44.369645 15764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:44 dist-test-slave-vl24 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194604, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:44.373296 15764 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:16:44.374754 15764 webserver.cc:460] Webserver started at http://127.5.83.126:35875/ using document root <none> and password file <none>
I20260923 20:16:44.375449 15764 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:44.375597 15764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:44.375931 15764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:44.378968 15764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "93c6cd1767814329bf305edf0f8db418"
format_stamp: "Formatted at 2026-09-23 20:16:44 on dist-test-slave-vl24"
server_key: "d6393cca69d8d84d634b5978fdacd9ff"
server_key_iv: "46b4a3f4d9fa700e9a1da6aa5094cdeb"
server_key_version: "encryptionkey@0"
I20260923 20:16:44.379724 15764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "93c6cd1767814329bf305edf0f8db418"
format_stamp: "Formatted at 2026-09-23 20:16:44 on dist-test-slave-vl24"
server_key: "d6393cca69d8d84d634b5978fdacd9ff"
server_key_iv: "46b4a3f4d9fa700e9a1da6aa5094cdeb"
server_key_version: "encryptionkey@0"
I20260923 20:16:44.384994 15764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:16:44.388358 15779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:44.389699 15764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:16:44.389895 15764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "93c6cd1767814329bf305edf0f8db418"
format_stamp: "Formatted at 2026-09-23 20:16:44 on dist-test-slave-vl24"
server_key: "d6393cca69d8d84d634b5978fdacd9ff"
server_key_iv: "46b4a3f4d9fa700e9a1da6aa5094cdeb"
server_key_version: "encryptionkey@0"
I20260923 20:16:44.390089 15764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:44.425200 15764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:44.429706 15764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:44.430043 15764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:44.448807 15764 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:35181
I20260923 20:16:44.448844 15830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:35181 every 8 connection(s)
I20260923 20:16:44.450261 15764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:16:44.454164 15831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:44.458873 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 15764
I20260923 20:16:44.459091 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:16:44.459913 5453 external_mini_cluster.cc:1428] Setting key fc1316e043f2f26749617352d786f3d5
I20260923 20:16:44.466619 15831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418: Bootstrap starting.
I20260923 20:16:44.470273 15831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:44.471477 15831 log.cc:824] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:16:44 dist-test-slave-vl24 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:16:44.475075 15831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418: No bootstrap required, opened a new log
I20260923 20:16:44.485020 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:44.462002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.5.83.126:35181)
0923 20:16:44.462760 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:44.462771 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:44.462811 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:16:44.464972 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:44.464997 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:44.465367 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:44.465696 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:44.467584 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:44.468936 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:44.470662 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:44.471330 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:44.471458 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:44.475981 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:44.475992 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:44.476011 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:44.476054 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:44.480181 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:44.481003 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:44.481017 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:44.481028 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:44.481253 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:44.481610 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:44.481615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:44.481619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:44.482421 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:44.482754 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:44.483059 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 20:16:44.483514 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":360,"threads_started":1}
I20260923 20:16:44.486189 15831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c6cd1767814329bf305edf0f8db418" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 35181 } }
I20260923 20:16:44.486580 15831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:44.486706 15831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c6cd1767814329bf305edf0f8db418, State: Initialized, Role: FOLLOWER
I20260923 20:16:44.487370 15831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c6cd1767814329bf305edf0f8db418" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 35181 } }
I20260923 20:16:44.487617 15831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:16:44.487756 15831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:16:44.487888 15831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:44.490185 15831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c6cd1767814329bf305edf0f8db418" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 35181 } }
I20260923 20:16:44.490688 15831 leader_election.cc:302] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93c6cd1767814329bf305edf0f8db418; no voters:
I20260923 20:16:44.491387 15831 leader_election.cc:288] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:16:44.491551 15836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:44.492267 15836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [term 1 LEADER]: Becoming Leader. State: Replica: 93c6cd1767814329bf305edf0f8db418, State: Running, Role: LEADER
I20260923 20:16:44.493055 15836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c6cd1767814329bf305edf0f8db418" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 35181 } }
I20260923 20:16:44.493898 15831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:16:44.496022 15838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93c6cd1767814329bf305edf0f8db418. Latest consensus state: current_term: 1 leader_uuid: "93c6cd1767814329bf305edf0f8db418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c6cd1767814329bf305edf0f8db418" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 35181 } } }
I20260923 20:16:44.496354 15838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:44.496714 15837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93c6cd1767814329bf305edf0f8db418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c6cd1767814329bf305edf0f8db418" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 35181 } } }
I20260923 20:16:44.497182 15837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418 [sys.catalog]: This master's current role is: LEADER
I20260923 20:16:44.507196 15842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:16:44.512475 15842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:16:44.521677 15842 catalog_manager.cc:1347] Generated new cluster ID: 68e771f7f4344dbebb33eee7131f6ddc
I20260923 20:16:44.521840 15842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:16:44.563328 15842 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:16:44.564695 15842 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:16:44.576709 15842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418: Generated new TSK 0
I20260923 20:16:44.577667 15842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:16:44.649820 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:35181
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:44.980237 15857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:44.980536 15857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:44.980724 15857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:44.987959 15857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:44.988126 15857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:44.988301 15857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:16:44.998620 15857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:35181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:44.999926 15857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:45.001293 15857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:45.014909 15862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:45.014926 15863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:45.015036 15865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:45.016084 15857 server_base.cc:1034] running on GCE node
I20260923 20:16:45.016796 15857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:45.017717 15857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:45.018945 15857 hybrid_clock.cc:648] HybridClock initialized: now 1790194605018898 us; error 40 us; skew 500 ppm
I20260923 20:16:45.019268 15857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:45 dist-test-slave-vl24 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194605, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:45.022549 15857 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:16:45.023926 15857 webserver.cc:460] Webserver started at http://127.5.83.65:43329/ using document root <none> and password file <none>
I20260923 20:16:45.024667 15857 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:45.024768 15857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:45.025023 15857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:45.027822 15857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "22e0a6c55bbb4fda80cbf3e3118db70c"
format_stamp: "Formatted at 2026-09-23 20:16:45 on dist-test-slave-vl24"
server_key: "15f015485e8418752805c6127b0163af"
server_key_iv: "04a39b7fe245c8a074b8866b4539732a"
server_key_version: "encryptionkey@0"
I20260923 20:16:45.028609 15857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "22e0a6c55bbb4fda80cbf3e3118db70c"
format_stamp: "Formatted at 2026-09-23 20:16:45 on dist-test-slave-vl24"
server_key: "15f015485e8418752805c6127b0163af"
server_key_iv: "04a39b7fe245c8a074b8866b4539732a"
server_key_version: "encryptionkey@0"
I20260923 20:16:45.033862 15857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:16:45.037215 15872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:45.038535 15857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:16:45.038694 15857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "22e0a6c55bbb4fda80cbf3e3118db70c"
format_stamp: "Formatted at 2026-09-23 20:16:45 on dist-test-slave-vl24"
server_key: "15f015485e8418752805c6127b0163af"
server_key_iv: "04a39b7fe245c8a074b8866b4539732a"
server_key_version: "encryptionkey@0"
I20260923 20:16:45.038880 15857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:45.055281 15857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:45.059633 15857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:45.059921 15857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:45.060901 15857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:45.062291 15857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:45.062363 15857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:45.062487 15857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:45.062551 15857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:45.096421 15857 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:42463
I20260923 20:16:45.096501 15984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:42463 every 8 connection(s)
I20260923 20:16:45.097755 15857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:16:45.106333 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 15857
I20260923 20:16:45.106556 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:16:45.107385 5453 external_mini_cluster.cc:1428] Setting key 3fda3f6274ae325f022fec38512b4985
Sep 23 20:16:45 dist-test-slave-vl24 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194605, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:16:45.120152 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:45.100889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:35181 (local address 127.5.83.65:46351)
0923 20:16:45.101579 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:45.101616 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:45.103317 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:45.104033 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:45.104041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:45.104684 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:45.105251 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:45.105268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:45.106323 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:45.106329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:45.108246 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:45.108259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:45.109203 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:45.109211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:45.109503 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:45.110222 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:45.110258 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:45.113018 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:45.116019 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:45.116029 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:45.116047 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:45.116490 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:45.116943 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:45.116948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:45.116952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:45.117150 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:45.117697 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:45.117704 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:45.118203 (+ 499us) client_negotiation.cc:769] Sending connection context
0923 20:16:45.118556 (+ 353us) client_negotiation.cc:240] Negotiation successful
0923 20:16:45.119103 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":280,"threads_started":1}
I20260923 20:16:45.121196 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:45.100246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:46351 (local address 127.5.83.126:35181)
0923 20:16:45.100743 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:45.100750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:45.101740 (+ 990us) server_negotiation.cc:407] Connection header received
0923 20:16:45.103598 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:45.103603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:45.103672 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:45.103779 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:45.105409 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:45.106154 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:45.108407 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:45.108958 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:45.109107 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:45.113255 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:45.113260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:45.113264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:45.113298 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:45.115856 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:45.116679 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:45.116687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:45.116695 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:45.116816 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:45.117276 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:45.117285 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:45.117287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:45.117559 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:45.117724 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:45.120747 (+ 3023us) server_negotiation.cc:299] Negotiation successful
0923 20:16:45.120969 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":218,"threads_started":1}
I20260923 20:16:45.122536 15985 heartbeater.cc:347] Connected to a master server at 127.5.83.126:35181
I20260923 20:16:45.122846 15985 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:45.123576 15985 heartbeater.cc:511] Master 127.5.83.126:35181 requested a full tablet report, sending...
I20260923 20:16:45.125669 15796 ts_manager.cc:194] Registered new tserver with Master: 22e0a6c55bbb4fda80cbf3e3118db70c (127.5.83.65:42463)
I20260923 20:16:45.127138 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:46351
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:16:45.163444 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:35181
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:45.510852 15991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:45.511130 15991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:45.511227 15991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:45.518615 15991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:45.518743 15991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:45.518857 15991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:16:45.528856 15991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:35181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:45.530239 15991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:45.531507 15991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:45.545010 15997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:45.545078 15996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:45.545043 15999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:45.545866 15991 server_base.cc:1034] running on GCE node
I20260923 20:16:45.546487 15991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:45.547421 15991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:45.548655 15991 hybrid_clock.cc:648] HybridClock initialized: now 1790194605548605 us; error 62 us; skew 500 ppm
I20260923 20:16:45.548990 15991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:45 dist-test-slave-vl24 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194605, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:45.552279 15991 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:16:45.553871 15991 webserver.cc:460] Webserver started at http://127.5.83.66:46179/ using document root <none> and password file <none>
I20260923 20:16:45.554584 15991 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:45.554729 15991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:45.555032 15991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:45.557991 15991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "45cf52b5294c46cd8c0aae10b285dc5c"
format_stamp: "Formatted at 2026-09-23 20:16:45 on dist-test-slave-vl24"
server_key: "32eeb173cd5032925e2fbb5851e3c960"
server_key_iv: "0d4bd876f27a513c32a17ba064134d65"
server_key_version: "encryptionkey@0"
I20260923 20:16:45.558763 15991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "45cf52b5294c46cd8c0aae10b285dc5c"
format_stamp: "Formatted at 2026-09-23 20:16:45 on dist-test-slave-vl24"
server_key: "32eeb173cd5032925e2fbb5851e3c960"
server_key_iv: "0d4bd876f27a513c32a17ba064134d65"
server_key_version: "encryptionkey@0"
I20260923 20:16:45.564049 15991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 20:16:45.567404 16006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:45.568722 15991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:16:45.568877 15991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "45cf52b5294c46cd8c0aae10b285dc5c"
format_stamp: "Formatted at 2026-09-23 20:16:45 on dist-test-slave-vl24"
server_key: "32eeb173cd5032925e2fbb5851e3c960"
server_key_iv: "0d4bd876f27a513c32a17ba064134d65"
server_key_version: "encryptionkey@0"
I20260923 20:16:45.569087 15991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:45.595516 15991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:45.599931 15991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:45.600235 15991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:45.601341 15991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:45.602720 15991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:45.602824 15991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:45.602969 15991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:45.603042 15991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:45.637503 15991 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:44831
I20260923 20:16:45.637495 16118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:44831 every 8 connection(s)
I20260923 20:16:45.639163 15991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:16:45.641542 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 15991
I20260923 20:16:45.641722 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:16:45.642217 5453 external_mini_cluster.cc:1428] Setting key 18c49b59e77a18b8740591727bc9e34a
Sep 23 20:16:45 dist-test-slave-vl24 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194605, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:16:45.660054 16122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:45.642942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:50683 (local address 127.5.83.126:35181)
0923 20:16:45.644633 (+ 1691us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:45.644637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:45.644654 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:16:45.645576 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:45.645585 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:45.645674 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:45.645795 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:45.647237 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:45.648361 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:45.649957 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:45.650692 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:45.650831 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:45.654405 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:45.654410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:45.654413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:45.654446 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:45.656983 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:45.657729 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:45.657740 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:45.657743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:45.657864 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:45.658287 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:45.658292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:45.658296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:45.658584 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:45.658741 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:45.659432 (+ 691us) server_negotiation.cc:299] Negotiation successful
0923 20:16:45.659662 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1545,"thread_start_us":1454,"threads_started":1}
I20260923 20:16:45.661098 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:45.642446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:35181 (local address 127.5.83.66:50683)
0923 20:16:45.643295 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:45.643341 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:45.645317 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:45.646012 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:45.646020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:45.646519 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:45.647069 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:45.647086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:45.648512 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:45.648515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:45.649808 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:45.649819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:45.650883 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:45.650887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:45.651083 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:45.651787 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:45.651808 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:45.654187 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:45.657120 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:45.657129 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:45.657159 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:45.657602 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:45.657986 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:45.657991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:45.657995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:45.658168 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:45.658710 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:45.658718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:45.659049 (+ 331us) client_negotiation.cc:769] Sending connection context
0923 20:16:45.659310 (+ 261us) client_negotiation.cc:240] Negotiation successful
0923 20:16:45.659923 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":353,"threads_started":1}
I20260923 20:16:45.662544 16119 heartbeater.cc:347] Connected to a master server at 127.5.83.126:35181
I20260923 20:16:45.662854 16119 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:45.663642 16119 heartbeater.cc:511] Master 127.5.83.126:35181 requested a full tablet report, sending...
I20260923 20:16:45.665086 15796 ts_manager.cc:194] Registered new tserver with Master: 45cf52b5294c46cd8c0aae10b285dc5c (127.5.83.66:44831)
I20260923 20:16:45.665895 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:50683
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:16:45.699522 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:35181
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:16:46.026173 16125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:16:46.026463 16125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:16:46.026657 16125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:16:46.034263 16125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:16:46.034372 16125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:16:46.034574 16125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:16:46.044477 16125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:35181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:16:46.045819 16125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:16:46.047170 16125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:16:46.060091 16133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:46.060215 16130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:16:46.060324 16131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:16:46.061010 16125 server_base.cc:1034] running on GCE node
I20260923 20:16:46.061597 16125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:16:46.062630 16125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:16:46.063858 16125 hybrid_clock.cc:648] HybridClock initialized: now 1790194606063795 us; error 54 us; skew 500 ppm
I20260923 20:16:46.064167 16125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:16:46 dist-test-slave-vl24 krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194606, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:16:46.067317 16125 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:16:46.068797 16125 webserver.cc:460] Webserver started at http://127.5.83.67:36621/ using document root <none> and password file <none>
I20260923 20:16:46.069517 16125 fs_manager.cc:359] Metadata directory not provided
I20260923 20:16:46.069648 16125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:16:46.069962 16125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:16:46.072938 16125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "d1131dc78cea4a64984640db6d52bfd4"
format_stamp: "Formatted at 2026-09-23 20:16:46 on dist-test-slave-vl24"
server_key: "76a6c0d4206c5bb09d519b8a2c7aefa8"
server_key_iv: "6bbf2697bc4c71afdda80e78370b9541"
server_key_version: "encryptionkey@0"
I20260923 20:16:46.073741 16125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "d1131dc78cea4a64984640db6d52bfd4"
format_stamp: "Formatted at 2026-09-23 20:16:46 on dist-test-slave-vl24"
server_key: "76a6c0d4206c5bb09d519b8a2c7aefa8"
server_key_iv: "6bbf2697bc4c71afdda80e78370b9541"
server_key_version: "encryptionkey@0"
I20260923 20:16:46.078788 16125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260923 20:16:46.082278 16140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:46.083593 16125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:16:46.083766 16125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "d1131dc78cea4a64984640db6d52bfd4"
format_stamp: "Formatted at 2026-09-23 20:16:46 on dist-test-slave-vl24"
server_key: "76a6c0d4206c5bb09d519b8a2c7aefa8"
server_key_iv: "6bbf2697bc4c71afdda80e78370b9541"
server_key_version: "encryptionkey@0"
I20260923 20:16:46.083971 16125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:16:46.120204 16125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:16:46.124397 16125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:16:46.124727 16125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:16:46.125710 16125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:16:46.127125 16125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:16:46.127197 16125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:46.127378 16125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:16:46.127432 16125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:16:46.130144 15985 heartbeater.cc:503] Master 127.5.83.126:35181 was elected leader, sending a full tablet report...
I20260923 20:16:46.162097 16125 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:38285
I20260923 20:16:46.162218 16252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:38285 every 8 connection(s)
I20260923 20:16:46.163419 16125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:16:46.164009 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 16125
I20260923 20:16:46.164201 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:16:46.164845 5453 external_mini_cluster.cc:1428] Setting key 5c8ceafe0a46719ab77bb1a00650c582
Sep 23 20:16:46 dist-test-slave-vl24 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194606, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:16:46.184018 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.166304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:56857 (local address 127.5.83.126:35181)
0923 20:16:46.166846 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:46.166857 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:46.167977 (+ 1120us) server_negotiation.cc:407] Connection header received
0923 20:16:46.169912 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:46.169917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:46.170006 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:46.170123 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:46.171571 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.172325 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.174082 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.174477 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.174609 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.178046 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:46.178054 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:46.178063 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:46.178095 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:46.181361 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:46.182036 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:46.182040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:46.182042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:46.182158 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:46.182597 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:46.182600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:46.182602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:46.182861 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:46.182975 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:46.183606 (+ 631us) server_negotiation.cc:299] Negotiation successful
0923 20:16:46.183761 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":236,"threads_started":1}
I20260923 20:16:46.185045 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.167029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:35181 (local address 127.5.83.67:56857)
0923 20:16:46.167818 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:46.167862 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:46.169651 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:46.170357 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:46.170365 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:46.170860 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:16:46.171393 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:46.171421 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.172490 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.172494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:46.173932 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:46.173946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.174697 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.174704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:46.174944 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.175616 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:16:46.175642 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:16:46.177851 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:16:46.181490 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:46.181496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:46.181508 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:46.181926 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:46.182281 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:16:46.182285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:16:46.182287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:16:46.182494 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:16:46.182970 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:16:46.182976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:16:46.183277 (+ 301us) client_negotiation.cc:769] Sending connection context
0923 20:16:46.183504 (+ 227us) client_negotiation.cc:240] Negotiation successful
0923 20:16:46.183866 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":344,"threads_started":1}
I20260923 20:16:46.186301 16253 heartbeater.cc:347] Connected to a master server at 127.5.83.126:35181
I20260923 20:16:46.186594 16253 heartbeater.cc:464] Registering TS with master...
I20260923 20:16:46.187402 16253 heartbeater.cc:511] Master 127.5.83.126:35181 requested a full tablet report, sending...
I20260923 20:16:46.188799 15796 ts_manager.cc:194] Registered new tserver with Master: d1131dc78cea4a64984640db6d52bfd4 (127.5.83.67:38285)
I20260923 20:16:46.189579 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:56857
I20260923 20:16:46.193042 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:16:46.210631 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.197590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.5.83.126:35181)
0923 20:16:46.197748 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:46.197752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:46.198299 (+ 547us) server_negotiation.cc:407] Connection header received
0923 20:16:46.198509 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:46.198513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:46.198575 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:46.198675 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:46.200041 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.200855 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.202052 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.202479 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.202623 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.205104 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:46.205109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:46.205112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:46.205145 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:46.207734 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:46.208553 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:46.208560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:46.208565 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:46.208711 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:46.209122 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:46.209128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:46.209131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:46.209475 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:46.209734 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:46.210088 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 20:16:46.210317 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:16:46.214846 15796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:16:46.221690 15796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:16:46.247359 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.234479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.5.83.67:38285)
0923 20:16:46.235426 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:46.235435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:46.236090 (+ 655us) server_negotiation.cc:407] Connection header received
0923 20:16:46.236587 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:46.236598 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:46.236917 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:46.237099 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:46.238200 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.240251 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.243778 (+ 3527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.245649 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.245808 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.246063 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:46.246836 (+ 773us) server_negotiation.cc:299] Negotiation successful
0923 20:16:46.247018 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":754,"threads_started":1}
I20260923 20:16:46.247453 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.233789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:38285 (local address 127.0.0.1:51238)
0923 20:16:46.235944 (+ 2155us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:46.235986 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:46.236259 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:46.237461 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:46.237467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:46.237495 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:46.237943 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:46.237960 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.240475 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.240483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:46.243519 (+ 3036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:46.243538 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.246238 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.246243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:46.246520 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.246573 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 20:16:46.246763 (+ 190us) client_negotiation.cc:240] Negotiation successful
0923 20:16:46.246959 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1907,"mutex_wait_us":2,"thread_start_us":10664,"threads_started":1}
I20260923 20:16:46.251094 16188 tablet_service.cc:1467] Processing CreateTablet for tablet c45add1fedbd4c54b500f0c52e42ac16 (DEFAULT_TABLE table=test-table [id=6209ed68f1644df3a07e00fc27590b87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:46.252691 16188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45add1fedbd4c54b500f0c52e42ac16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:46.254084 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.235058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.5.83.66:44831)
0923 20:16:46.243922 (+ 8864us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:46.243941 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:46.245083 (+ 1142us) server_negotiation.cc:407] Connection header received
0923 20:16:46.245180 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:46.245187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:46.245457 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:46.246322 (+ 865us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:46.246717 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.248982 (+ 2265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.251562 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.253042 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.253174 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.253347 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:46.253599 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 20:16:46.253756 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8692,"thread_start_us":8542,"threads_started":1}
I20260923 20:16:46.254101 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.234139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:44831 (local address 127.0.0.1:51010)
0923 20:16:46.244812 (+ 10673us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:46.244832 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:46.244957 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:46.245676 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:46.245684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:46.245718 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:46.246561 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:46.246572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.249108 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.249116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:46.251356 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:46.251370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.253244 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.253250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:46.253408 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.253440 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:16:46.253547 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 20:16:46.253678 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10511,"mutex_wait_us":346,"thread_start_us":3086,"threads_started":1}
I20260923 20:16:46.254379 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.235772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.5.83.65:42463)
0923 20:16:46.244282 (+ 8510us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:46.244292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:46.246329 (+ 2037us) server_negotiation.cc:407] Connection header received
0923 20:16:46.246492 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:46.246498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:46.246726 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:46.246904 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:46.247807 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.249248 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.251709 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.253065 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.253174 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.253382 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:46.253886 (+ 504us) server_negotiation.cc:299] Negotiation successful
0923 20:16:46.254037 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8308,"thread_start_us":8133,"threads_started":1}
I20260923 20:16:46.255064 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.235143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:42463 (local address 127.0.0.1:38588)
0923 20:16:46.246166 (+ 11023us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:46.246206 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:46.246332 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:46.247176 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:46.247181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:46.247203 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:46.247596 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:46.247610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.249400 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.249405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:46.251546 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:46.251556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.253499 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.253503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:46.253684 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.253718 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 20:16:46.254568 (+ 850us) client_negotiation.cc:240] Negotiation successful
0923 20:16:46.254723 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10878,"mutex_wait_us":1,"thread_start_us":10696,"threads_started":1}
I20260923 20:16:46.257613 16054 tablet_service.cc:1467] Processing CreateTablet for tablet c45add1fedbd4c54b500f0c52e42ac16 (DEFAULT_TABLE table=test-table [id=6209ed68f1644df3a07e00fc27590b87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:46.258690 16054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45add1fedbd4c54b500f0c52e42ac16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:46.258314 15920 tablet_service.cc:1467] Processing CreateTablet for tablet c45add1fedbd4c54b500f0c52e42ac16 (DEFAULT_TABLE table=test-table [id=6209ed68f1644df3a07e00fc27590b87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:16:46.259651 15920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45add1fedbd4c54b500f0c52e42ac16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:16:46.269870 16272 tablet_bootstrap.cc:492] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c: Bootstrap starting.
I20260923 20:16:46.273777 16272 tablet_bootstrap.cc:654] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:46.275121 16272 log.cc:824] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:46.276595 16273 tablet_bootstrap.cc:492] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4: Bootstrap starting.
I20260923 20:16:46.278344 16272 tablet_bootstrap.cc:492] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c: No bootstrap required, opened a new log
I20260923 20:16:46.278391 16274 tablet_bootstrap.cc:492] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c: Bootstrap starting.
I20260923 20:16:46.278716 16272 ts_tablet_manager.cc:1397] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 20:16:46.280314 16273 tablet_bootstrap.cc:654] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:46.281154 16274 tablet_bootstrap.cc:654] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c: Neither blocks nor log segments found. Creating new log.
I20260923 20:16:46.281559 16273 log.cc:824] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:46.282193 16274 log.cc:824] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:16:46.284699 16273 tablet_bootstrap.cc:492] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4: No bootstrap required, opened a new log
I20260923 20:16:46.284956 16273 ts_tablet_manager.cc:1397] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260923 20:16:46.286655 16274 tablet_bootstrap.cc:492] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c: No bootstrap required, opened a new log
I20260923 20:16:46.286903 16274 ts_tablet_manager.cc:1397] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260923 20:16:46.290324 16272 raft_consensus.cc:348] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1131dc78cea4a64984640db6d52bfd4" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38285 } } peers { permanent_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 44831 } } peers { permanent_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 42463 } }
I20260923 20:16:46.290793 16272 raft_consensus.cc:374] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:46.290930 16272 raft_consensus.cc:729] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cf52b5294c46cd8c0aae10b285dc5c, State: Initialized, Role: FOLLOWER
I20260923 20:16:46.291570 16272 consensus_queue.cc:260] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1131dc78cea4a64984640db6d52bfd4" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38285 } } peers { permanent_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 44831 } } peers { permanent_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 42463 } }
I20260923 20:16:46.292739 16119 heartbeater.cc:503] Master 127.5.83.126:35181 was elected leader, sending a full tablet report...
I20260923 20:16:46.293596 16272 ts_tablet_manager.cc:1428] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260923 20:16:46.295481 16273 raft_consensus.cc:348] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1131dc78cea4a64984640db6d52bfd4" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38285 } } peers { permanent_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 44831 } } peers { permanent_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 42463 } }
I20260923 20:16:46.295861 16273 raft_consensus.cc:374] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:46.295982 16273 raft_consensus.cc:729] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1131dc78cea4a64984640db6d52bfd4, State: Initialized, Role: FOLLOWER
I20260923 20:16:46.296646 16273 consensus_queue.cc:260] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1131dc78cea4a64984640db6d52bfd4" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38285 } } peers { permanent_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 44831 } } peers { permanent_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 42463 } }
I20260923 20:16:46.297763 16253 heartbeater.cc:503] Master 127.5.83.126:35181 was elected leader, sending a full tablet report...
I20260923 20:16:46.297955 16274 raft_consensus.cc:348] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1131dc78cea4a64984640db6d52bfd4" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38285 } } peers { permanent_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 44831 } } peers { permanent_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 42463 } }
I20260923 20:16:46.298336 16273 ts_tablet_manager.cc:1428] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260923 20:16:46.298390 16274 raft_consensus.cc:374] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:16:46.298484 16274 raft_consensus.cc:729] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22e0a6c55bbb4fda80cbf3e3118db70c, State: Initialized, Role: FOLLOWER
I20260923 20:16:46.299161 16274 consensus_queue.cc:260] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1131dc78cea4a64984640db6d52bfd4" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38285 } } peers { permanent_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 44831 } } peers { permanent_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 42463 } }
I20260923 20:16:46.300890 16274 ts_tablet_manager.cc:1428] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260923 20:16:46.309547 16278 raft_consensus.cc:482] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:16:46.309796 16278 raft_consensus.cc:504] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1131dc78cea4a64984640db6d52bfd4" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38285 } } peers { permanent_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 44831 } } peers { permanent_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 42463 } }
I20260923 20:16:46.311233 16278 leader_election.cc:288] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1131dc78cea4a64984640db6d52bfd4 (127.5.83.67:38285), 22e0a6c55bbb4fda80cbf3e3118db70c (127.5.83.65:42463)
I20260923 20:16:46.320611 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.311712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:51131 (local address 127.5.83.65:42463)
0923 20:16:46.311902 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:46.311909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:46.312598 (+ 689us) server_negotiation.cc:407] Connection header received
0923 20:16:46.312821 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:46.312827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:46.312901 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:46.313001 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:46.313868 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.315005 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.318173 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.319438 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.319578 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.319687 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:46.320166 (+ 479us) server_negotiation.cc:299] Negotiation successful
0923 20:16:46.320325 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:16:46.320611 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.311730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:60507 (local address 127.5.83.67:38285)
0923 20:16:46.311902 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:46.311908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:46.312621 (+ 713us) server_negotiation.cc:407] Connection header received
0923 20:16:46.312821 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:46.312827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:46.312908 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:46.313021 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:16:46.313868 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.315018 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.318173 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.319438 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.319578 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.319687 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:46.320166 (+ 479us) server_negotiation.cc:299] Negotiation successful
0923 20:16:46.320318 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:16:46.320864 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.311593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:42463 (local address 127.5.83.66:51131)
0923 20:16:46.312489 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:46.312503 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:46.312682 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:46.313037 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:46.313042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:46.313075 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:46.313732 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:46.313740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.315153 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.315157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:46.317995 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:46.318004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.319700 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.319704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:46.319938 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.320002 (+ 64us) client_negotiation.cc:769] Sending connection context
0923 20:16:46.320088 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 20:16:46.320265 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":327,"spinlock_wait_cycles":22784,"thread_start_us":490,"threads_started":1}
I20260923 20:16:46.321004 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.311478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:38285 (local address 127.5.83.66:60507)
0923 20:16:46.312469 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0923 20:16:46.312504 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 20:16:46.312682 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:16:46.313037 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:16:46.313042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:16:46.313072 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:16:46.313704 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:46.313723 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.315152 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.315157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:46.317979 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:16:46.317997 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.319697 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.319703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:16:46.319942 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.319991 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 20:16:46.320088 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:16:46.320265 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":331,"spinlock_wait_cycles":21888,"thread_start_us":639,"threads_started":1}
I20260923 20:16:46.321933 16208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45add1fedbd4c54b500f0c52e42ac16" candidate_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1131dc78cea4a64984640db6d52bfd4" is_pre_election: true
I20260923 20:16:46.321972 15940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45add1fedbd4c54b500f0c52e42ac16" candidate_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c" is_pre_election: true
I20260923 20:16:46.322461 16208 raft_consensus.cc:2454] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45cf52b5294c46cd8c0aae10b285dc5c in term 0.
I20260923 20:16:46.322537 15940 raft_consensus.cc:2454] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45cf52b5294c46cd8c0aae10b285dc5c in term 0.
I20260923 20:16:46.323262 16010 leader_election.cc:302] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22e0a6c55bbb4fda80cbf3e3118db70c, 45cf52b5294c46cd8c0aae10b285dc5c, d1131dc78cea4a64984640db6d52bfd4; no voters:
I20260923 20:16:46.323737 16278 raft_consensus.cc:2781] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:16:46.323876 16278 raft_consensus.cc:482] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:16:46.323987 16278 raft_consensus.cc:3037] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:46.326234 16278 raft_consensus.cc:504] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1131dc78cea4a64984640db6d52bfd4" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38285 } } peers { permanent_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 44831 } } peers { permanent_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 42463 } }
I20260923 20:16:46.326915 16278 leader_election.cc:288] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [CANDIDATE]: Term 1 election: Requested vote from peers d1131dc78cea4a64984640db6d52bfd4 (127.5.83.67:38285), 22e0a6c55bbb4fda80cbf3e3118db70c (127.5.83.65:42463)
I20260923 20:16:46.327329 16208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45add1fedbd4c54b500f0c52e42ac16" candidate_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1131dc78cea4a64984640db6d52bfd4"
I20260923 20:16:46.327385 15940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45add1fedbd4c54b500f0c52e42ac16" candidate_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c"
I20260923 20:16:46.327554 16208 raft_consensus.cc:3037] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:46.327627 15940 raft_consensus.cc:3037] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:16:46.329818 16208 raft_consensus.cc:2454] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45cf52b5294c46cd8c0aae10b285dc5c in term 1.
I20260923 20:16:46.329917 15940 raft_consensus.cc:2454] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45cf52b5294c46cd8c0aae10b285dc5c in term 1.
I20260923 20:16:46.330220 16010 leader_election.cc:302] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45cf52b5294c46cd8c0aae10b285dc5c, d1131dc78cea4a64984640db6d52bfd4; no voters:
I20260923 20:16:46.330530 16278 raft_consensus.cc:2781] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:16:46.330999 16278 raft_consensus.cc:686] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 1 LEADER]: Becoming Leader. State: Replica: 45cf52b5294c46cd8c0aae10b285dc5c, State: Running, Role: LEADER
I20260923 20:16:46.331460 16278 consensus_queue.cc:237] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1131dc78cea4a64984640db6d52bfd4" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38285 } } peers { permanent_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 44831 } } peers { permanent_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 42463 } }
I20260923 20:16:46.335850 15795 catalog_manager.cc:5495] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45cf52b5294c46cd8c0aae10b285dc5c (127.5.83.66). New cstate: current_term: 1 leader_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1131dc78cea4a64984640db6d52bfd4" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45cf52b5294c46cd8c0aae10b285dc5c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 42463 } health_report { overall_health: UNKNOWN } } }
W20260923 20:16:46.350158 15986 tablet.cc:2369] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:16:46.358021 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.351759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.5.83.66:44831)
0923 20:16:46.351947 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:46.351952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:46.351967 (+ 15us) server_negotiation.cc:407] Connection header received
0923 20:16:46.352112 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:46.352119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:46.352216 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:46.352379 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:16:46.352797 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.353717 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.355570 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.356083 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.356239 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.356911 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:16:46.357420 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:16:46.357541 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:46.357683 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 20:16:46.357744 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:16:46.366212 15940 raft_consensus.cc:1261] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c [term 1 FOLLOWER]: Refusing update from remote peer 45cf52b5294c46cd8c0aae10b285dc5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:46.366214 16208 raft_consensus.cc:1261] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4 [term 1 FOLLOWER]: Refusing update from remote peer 45cf52b5294c46cd8c0aae10b285dc5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:16:46.367156 16283 consensus_queue.cc:1035] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1131dc78cea4a64984640db6d52bfd4" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:46.367477 16278 consensus_queue.cc:1035] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22e0a6c55bbb4fda80cbf3e3118db70c" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:16:46.378643 16286 mvcc.cc:205] Tried to move back new op lower bound from 7332637107666595840 to 7332637107537539072. Current Snapshot: MvccSnapshot[applied={T|T < 7332637107666595840}]
I20260923 20:16:46.380182 16288 mvcc.cc:205] Tried to move back new op lower bound from 7332637107666595840 to 7332637107537539072. Current Snapshot: MvccSnapshot[applied={T|T < 7332637107666595840}]
I20260923 20:16:46.396350 15795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:46.396683 15795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:16:46.400225 15795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93c6cd1767814329bf305edf0f8db418: Sending DeleteTablet for 3 replicas of tablet c45add1fedbd4c54b500f0c52e42ac16
I20260923 20:16:46.401508 16054 tablet_service.cc:1514] Processing DeleteTablet for tablet c45add1fedbd4c54b500f0c52e42ac16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:46 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:51010
I20260923 20:16:46.401526 16188 tablet_service.cc:1514] Processing DeleteTablet for tablet c45add1fedbd4c54b500f0c52e42ac16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:46 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:51238
I20260923 20:16:46.401925 15920 tablet_service.cc:1514] Processing DeleteTablet for tablet c45add1fedbd4c54b500f0c52e42ac16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:16:46 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:38588
I20260923 20:16:46.404367 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 15857
I20260923 20:16:46.414690 16296 tablet_replica.cc:330] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c: stopping tablet replica
I20260923 20:16:46.415225 16296 raft_consensus.cc:2229] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:16:46.415939 16296 raft_consensus.cc:2258] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:46.416764 16298 tablet_replica.cc:330] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c: stopping tablet replica
I20260923 20:16:46.416826 16297 tablet_replica.cc:330] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4: stopping tablet replica
I20260923 20:16:46.417240 16298 raft_consensus.cc:2229] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:46.417260 16297 raft_consensus.cc:2229] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:16:46.417739 16297 raft_consensus.cc:2258] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:46.417898 16298 raft_consensus.cc:2258] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:16:46.419943 16297 ts_tablet_manager.cc:1905] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:46.420388 16298 ts_tablet_manager.cc:1905] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:46.426896 16296 ts_tablet_manager.cc:1905] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:16:46.427069 16297 ts_tablet_manager.cc:1918] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:46.427294 16297 log.cc:1196] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/c45add1fedbd4c54b500f0c52e42ac16
I20260923 20:16:46.427935 16297 ts_tablet_manager.cc:1939] T c45add1fedbd4c54b500f0c52e42ac16 P d1131dc78cea4a64984640db6d52bfd4: Deleting consensus metadata
I20260923 20:16:46.429916 15783 catalog_manager.cc:4841] TS d1131dc78cea4a64984640db6d52bfd4 (127.5.83.67:38285): tablet c45add1fedbd4c54b500f0c52e42ac16 (table test-table [id=6209ed68f1644df3a07e00fc27590b87]) successfully deleted
Sep 23 20:16:46 dist-test-slave-vl24 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:16:46.434437 16296 ts_tablet_manager.cc:1918] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:46.434639 16296 log.cc:1196] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/c45add1fedbd4c54b500f0c52e42ac16
I20260923 20:16:46.435324 16296 ts_tablet_manager.cc:1939] T c45add1fedbd4c54b500f0c52e42ac16 P 45cf52b5294c46cd8c0aae10b285dc5c: Deleting consensus metadata
I20260923 20:16:46.438251 16298 ts_tablet_manager.cc:1918] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:16:46.438441 16298 log.cc:1196] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/c45add1fedbd4c54b500f0c52e42ac16
I20260923 20:16:46.438818 15781 catalog_manager.cc:4841] TS 45cf52b5294c46cd8c0aae10b285dc5c (127.5.83.66:44831): tablet c45add1fedbd4c54b500f0c52e42ac16 (table test-table [id=6209ed68f1644df3a07e00fc27590b87]) successfully deleted
I20260923 20:16:46.439049 16298 ts_tablet_manager.cc:1939] T c45add1fedbd4c54b500f0c52e42ac16 P 22e0a6c55bbb4fda80cbf3e3118db70c: Deleting consensus metadata
I20260923 20:16:46.440805 15782 catalog_manager.cc:4841] TS 22e0a6c55bbb4fda80cbf3e3118db70c (127.5.83.65:42463): tablet c45add1fedbd4c54b500f0c52e42ac16 (table test-table [id=6209ed68f1644df3a07e00fc27590b87]) successfully deleted
I20260923 20:16:46.442276 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.405732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.5.83.65:42463)
0923 20:16:46.405957 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:46.405966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:46.421909 (+ 15943us) server_negotiation.cc:407] Connection header received
0923 20:16:46.422411 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:46.422417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:46.422520 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:46.422598 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:46.424322 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.425530 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.427341 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.427960 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.428056 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.434083 (+ 6027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:46.434100 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:46.434120 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:46.434168 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:46.438508 (+ 4340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:46.439420 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:46.439430 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:46.439433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:46.439618 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:46.440054 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:46.440059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:46.440063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:46.440520 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:46.441216 (+ 696us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:46.441706 (+ 490us) server_negotiation.cc:299] Negotiation successful
0923 20:16:46.441917 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:16:46.442920 15980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:46.545099 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 15857
I20260923 20:16:46.563932 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 15991
Sep 23 20:16:46 dist-test-slave-vl24 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:16:46.582703 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.565003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51032 (local address 127.5.83.66:44831)
0923 20:16:46.565171 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:46.565175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:46.565195 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:16:46.565250 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:46.565254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:46.565332 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:46.565420 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:46.567042 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.567834 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.569755 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.570186 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.570291 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.575298 (+ 5007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:46.575312 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:46.575331 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:46.575399 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:46.579552 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:46.580317 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:46.580326 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:46.580330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:46.580508 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:46.580918 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:46.580923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:46.580927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:46.581396 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:46.581719 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:46.582038 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 20:16:46.582277 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:16:46.583812 16114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:46.701455 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 15991
I20260923 20:16:46.720711 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 16125
Sep 23 20:16:46 dist-test-slave-vl24 krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:16:46.736256 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:16:46.721496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.5.83.67:38285)
0923 20:16:46.721689 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:16:46.721697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:16:46.721721 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:16:46.721929 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:16:46.721934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:16:46.722020 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:16:46.722144 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:16:46.723387 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.724126 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.725507 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:16:46.725911 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:16:46.726045 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:16:46.730951 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:16:46.730965 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:16:46.730980 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:16:46.731014 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:16:46.733892 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:46.734492 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:46.734507 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:46.734509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:46.734645 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:16:46.734956 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:16:46.734959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:16:46.734960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:16:46.735289 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:16:46.735511 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 20:16:46.735728 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 20:16:46.735902 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:16:46.737025 16248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:46.826259 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 16125
I20260923 20:16:46.843590 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 15764
I20260923 20:16:46.844738 15826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:16:46.955618 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 15764
2026-09-23T20:16:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4369 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:16:47 dist-test-slave-vl24 krb5kdc[16306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:16:47 dist-test-slave-vl24 krb5kdc[16306](info): set up 2 sockets
Sep 23 20:16:47 dist-test-slave-vl24 krb5kdc[16306](info): commencing operation
krb5kdc: starting...
W20260923 20:16:48.066994 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:16:48 dist-test-slave-vl24 krb5kdc[16306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:16:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/rangeradmin_127.5.83.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/rangeradmin_127.5.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/rangerlookup_127.5.83.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/rangerlookup_127.5.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/HTTP_127.5.83.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/HTTP_127.5.83.84@KRBTEST.COM.keytab.
I20260923 20:16:48.424141 5453 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCNgSr1/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/postgres -l logfile start
2026-09-23 20:16:50.993 UTC [16336] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 20:16:50.993 UTC [16336] LOG: listening on IPv4 address "127.5.83.84", port 42371
2026-09-23 20:16:50.997 UTC [16336] LOG: listening on Unix socket "/tmp/.s.PGSQL.42371"
2026-09-23 20:16:51.025 UTC [16339] LOG: database system was shut down at 2026-09-23 20:16:49 UTC
2026-09-23 20:16:51.029 UTC [16336] LOG: database system is ready to accept connections
I20260923 20:16:52.094461 5453 mini_postgres.cc:96] Postgres bound to 42371
2026-09-23 20:16:52.114 UTC [16347] FATAL: database "slave" does not exist
127.5.83.84:42371 - accepting connections
I20260923 20:16:52.115460 5453 mini_ranger.cc:161] Starting Ranger...
I20260923 20:16:52.144697 5453 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 20:16:52.636963 5453 mini_ranger.cc:87] Created ranger Postgres database
I20260923 20:16:52.637145 5453 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-admin
2026-09-23 20:16:52,993 [I] DB FLAVOR :POSTGRES
2026-09-23 20:16:52,993 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:16:52,994 [I] Checking connection..
2026-09-23 20:16:52,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 20:16:53,275 [I] Checking connection passed.
2026-09-23 20:16:53,275 [I] --------- Verifying version history table ---------
2026-09-23 20:16:53,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:16:53,530 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:16:53,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:16:53,781 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:16:53,781 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 20:16:53,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 20:16:54.026 UTC [16411] WARNING: there is no transaction in progress
2026-09-23 20:16:54,035 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 20:16:54,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:16:54,316 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 20:16:54,317 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:16:54,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 20:16:54,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 20:16:54,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vl24.c.gcp-upstream.internal','N') ;"
2026-09-23 20:16:55,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:16:55,307 [I] Table x_portal_user does not exist in database ranger
2026-09-23 20:16:55,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:16:55,542 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 20:16:55,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:16:55,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:16:56,056 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 20:16:56,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 20:16:57.041 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.059 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.076 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.091 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.111 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.131 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.198 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.205 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.219 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.229 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.240 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.253 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.263 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.268 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.275 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57.282 UTC [16519] WARNING: there is no transaction in progress
2026-09-23 20:16:57,911 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 20:16:57,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 20:16:58,163 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 20:16:58,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:16:58,398 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 20:16:58,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:16:58,646 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 20:16:58,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:16:58,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:16:59,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vl24.c.gcp-upstream.internal';"
2026-09-23 20:16:59,395 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 20:16:59,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 20:16:59,649 [I] DB_PATCHES have already been applied
I20260923 20:16:59.655717 5453 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCNgSr1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 20:16:59.655891 5453 mini_ranger.cc:193] Using host: 127.5.83.84
I20260923 20:16:59.673689 5453 mini_ranger.cc:227] Ranger admin URL: http://127.5.83.84:47301
Sep 23, 2026 8:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 8:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 8:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 8:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 8:17:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 20:17:00.826371 5453 mini_ranger.cc:160] Time spent starting Ranger: real 8.711s user 0.003s sys 0.048s
Sep 23, 2026 8:17:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 8:17:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 23, 2026 8:17:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 8:17:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 8:17:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 8:17:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 20:17:22.432758 5453 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/rangerkms_127.5.83.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/rangerkms_127.5.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.83.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.83.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.83.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/HTTP_127.5.83.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.83.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/HTTP_127.5.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 20:17:22 dist-test-slave-vl24 krb5kdc[16306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194642, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 20:17:22.640467 5453 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 20:17:22.664269 5453 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 20:17:23.552587 5453 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 20:17:23.552721 5453 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-kms
I20260923 20:17:23.552875 5453 mini_ranger_kms.cc:227] Using postgres at 127.5.83.84:42371
2026-09-23 20:17:24,021 [I] DB FLAVOR :POSTGRES
2026-09-23 20:17:24,021 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:17:24,021 [I] Checking connection
2026-09-23 20:17:24,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 20:17:24,273 [I] connection success
2026-09-23 20:17:24,273 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 20:17:24,273 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 20:17:24,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 20:17:24,506 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 20:17:24,506 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:17:24,506 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 20:17:24,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:42371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 20:17:24,759 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 20:17:25.250591 5453 mini_ranger_kms.cc:310] Created kms service
I20260923 20:17:25.442631 5453 mini_ranger_kms.cc:326] Created kudu user
I20260923 20:17:25.494278 5453 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 20:17:25 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194642, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:17:25.955763 5453 mini_ranger_kms.cc:384] Added ranger policy
I20260923 20:17:25.956094 5453 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 20:17:25.956151 5453 mini_ranger_kms.cc:240] Using host: 127.5.83.84
I20260923 20:17:25.978894 5453 mini_ranger_kms.cc:276] Ranger KMS PID: 16699
I20260923 20:17:25.979102 5453 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.83.84:40489
Sep 23, 2026 8:17:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:17:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:17:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 8:17:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 8:17:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 8:17:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 20:17:28.356415 5453 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.716s user 0.137s sys 0.363s
I20260923 20:17:28.356731 5453 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 20:17:28.356949 5453 mini_ranger_kms.cc:401] 127.5.83.84:40489/kms/v1/keys
Sep 23 20:17:28 dist-test-slave-vl24 krb5kdc[16306](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790194648, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.83.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 20:17:28 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194648, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.83.84@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
Sep 23 20:17:29 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194648, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.83.84@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
Sep 23 20:17:31 dist-test-slave-vl24 krb5kdc[16306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:17:31.529167 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:46359
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:46303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:46359
--ranger_config_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.84:40489/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:17:31.899672 16750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:17:31.900014 16750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:17:31.900174 16750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:17:31.907708 16750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:17:31.907882 16750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:17:31.907989 16750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:17:31.908087 16750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:17:31.908165 16750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:17:31.918856 16750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:46303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.83.84:40489/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:46359
--ranger_config_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:46359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:17:31.920260 16750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:17:31.921658 16750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:17:31.934314 16755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:17:31.934260 16756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:17:31.934841 16758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:17:31.935737 16750 server_base.cc:1034] running on GCE node
I20260923 20:17:31.936731 16750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:17:31.938544 16750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:17:31.939816 16750 hybrid_clock.cc:648] HybridClock initialized: now 1790194651939769 us; error 54 us; skew 500 ppm
I20260923 20:17:31.940233 16750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:17:31 dist-test-slave-vl24 krb5kdc[16306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194651, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:17:31.949776 16750 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:17:31.951573 16750 webserver.cc:460] Webserver started at http://127.5.83.126:43553/ using document root <none> and password file <none>
I20260923 20:17:31.952261 16750 fs_manager.cc:359] Metadata directory not provided
I20260923 20:17:31.952394 16750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:17:31.952829 16750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:17:31 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194651, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:17:32.011714 16750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "f644863bfb7347cfbf09e363c4c84093"
format_stamp: "Formatted at 2026-09-23 20:17:32 on dist-test-slave-vl24"
server_key: "f7274c6d040aa59e1a3ded562209ce0e"
server_key_iv: "b6a03a131927a3bfb3e7fb0286b00664"
server_key_version: "kuduclusterkey@0"
I20260923 20:17:32.012858 16750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "f644863bfb7347cfbf09e363c4c84093"
format_stamp: "Formatted at 2026-09-23 20:17:32 on dist-test-slave-vl24"
server_key: "f7274c6d040aa59e1a3ded562209ce0e"
server_key_iv: "b6a03a131927a3bfb3e7fb0286b00664"
server_key_version: "kuduclusterkey@0"
I20260923 20:17:32.019891 16750 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260923 20:17:32.041111 16765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:17:32.043038 16750 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 20:17:32.043259 16750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "f644863bfb7347cfbf09e363c4c84093"
format_stamp: "Formatted at 2026-09-23 20:17:32 on dist-test-slave-vl24"
server_key: "f7274c6d040aa59e1a3ded562209ce0e"
server_key_iv: "b6a03a131927a3bfb3e7fb0286b00664"
server_key_version: "kuduclusterkey@0"
I20260923 20:17:32.043533 16750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:17:32.062592 16750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:17:32.066423 16750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:17:32.066675 16750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:17:32.086380 16750 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:46359
I20260923 20:17:32.086423 16816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:46359 every 8 connection(s)
I20260923 20:17:32.087728 16750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:17:32.091334 16817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:17:32.093752 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 16750
I20260923 20:17:32.093963 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:17:32.094606 5453 external_mini_cluster.cc:1428] Setting key dd0d66472e208fb43017c77c0823e424
I20260923 20:17:32.102680 16817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093: Bootstrap starting.
I20260923 20:17:32.105988 16817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093: Neither blocks nor log segments found. Creating new log.
I20260923 20:17:32.106995 16817 log.cc:824] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:17:32 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:17:32.110167 16817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093: No bootstrap required, opened a new log
I20260923 20:17:32.118443 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:32.096622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.5.83.126:46359)
0923 20:17:32.097396 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:32.097408 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:32.097447 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:17:32.099332 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:32.099373 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:32.099781 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:32.100174 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:17:32.101513 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:32.102883 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:32.104113 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:32.104843 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:32.105038 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:32.109021 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:17:32.109031 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:17:32.109055 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:17:32.109095 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:17:32.113041 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:32.113981 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:32.114010 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:32.114018 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:32.114218 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:32.114708 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:32.114714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:32.114718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:32.115606 (+ 888us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:17:32.115893 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:32.116611 (+ 718us) server_negotiation.cc:299] Negotiation successful
0923 20:17:32.117048 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":352,"threads_started":1}
I20260923 20:17:32.122076 16817 raft_consensus.cc:348] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f644863bfb7347cfbf09e363c4c84093" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46359 } }
I20260923 20:17:32.122409 16817 raft_consensus.cc:374] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:17:32.122481 16817 raft_consensus.cc:729] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f644863bfb7347cfbf09e363c4c84093, State: Initialized, Role: FOLLOWER
I20260923 20:17:32.122926 16817 consensus_queue.cc:260] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f644863bfb7347cfbf09e363c4c84093" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46359 } }
I20260923 20:17:32.123119 16817 raft_consensus.cc:388] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:17:32.123214 16817 raft_consensus.cc:482] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:17:32.123320 16817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:17:32.125396 16817 raft_consensus.cc:504] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f644863bfb7347cfbf09e363c4c84093" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46359 } }
I20260923 20:17:32.125792 16817 leader_election.cc:302] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f644863bfb7347cfbf09e363c4c84093; no voters:
I20260923 20:17:32.126420 16817 leader_election.cc:288] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:17:32.126593 16822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:17:32.128778 16817 sys_catalog.cc:564] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:17:32.129040 16822 raft_consensus.cc:686] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [term 1 LEADER]: Becoming Leader. State: Replica: f644863bfb7347cfbf09e363c4c84093, State: Running, Role: LEADER
I20260923 20:17:32.129534 16822 consensus_queue.cc:237] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f644863bfb7347cfbf09e363c4c84093" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46359 } }
I20260923 20:17:32.132517 16824 sys_catalog.cc:455] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f644863bfb7347cfbf09e363c4c84093. Latest consensus state: current_term: 1 leader_uuid: "f644863bfb7347cfbf09e363c4c84093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f644863bfb7347cfbf09e363c4c84093" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46359 } } }
I20260923 20:17:32.133038 16824 sys_catalog.cc:458] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [sys.catalog]: This master's current role is: LEADER
I20260923 20:17:32.141445 16823 sys_catalog.cc:455] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f644863bfb7347cfbf09e363c4c84093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f644863bfb7347cfbf09e363c4c84093" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46359 } } }
I20260923 20:17:32.141882 16823 sys_catalog.cc:458] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093 [sys.catalog]: This master's current role is: LEADER
I20260923 20:17:32.141995 16833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:17:32.146306 16833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:17:32.147178 16817 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 20:17:32.167624 16833 catalog_manager.cc:1347] Generated new cluster ID: 7d8285e53c544e93abf6b47d6e628233
I20260923 20:17:32.167801 16833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:17:32.219192 16833 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:17:32.220685 16833 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:17:32.239398 16833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093: Generated new TSK 0
I20260923 20:17:32.240365 16833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:17:32.989364 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.84:40489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46359
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:46303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:17:33 dist-test-slave-vl24 krb5kdc[16306](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790194653, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 20:17:33.433303 16862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:17:33.433621 16862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:17:33.433787 16862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:17:33.444098 16862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:17:33.444242 16862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:17:33.444586 16862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:17:33.460712 16862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:46303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.83.84:40489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:17:33.462419 16862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:17:33.464140 16862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:17:33.483341 16872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:17:33.483775 16873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:17:33.488072 16862 server_base.cc:1034] running on GCE node
W20260923 20:17:33.488879 16875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:17:33.489769 16862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:17:33.495030 16862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:17:33.496377 16862 hybrid_clock.cc:648] HybridClock initialized: now 1790194653496296 us; error 65 us; skew 500 ppm
I20260923 20:17:33.496877 16862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:17:33 dist-test-slave-vl24 krb5kdc[16306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194653, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:17:33.501605 16862 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:17:33.507483 16862 webserver.cc:460] Webserver started at http://127.5.83.65:35939/ using document root <none> and password file <none>
I20260923 20:17:33.508556 16862 fs_manager.cc:359] Metadata directory not provided
I20260923 20:17:33.508760 16862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:17:33.509207 16862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:17:33 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194653, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:17:33.540104 16862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "ee2583c7bb984b5890b280896a01dfe9"
format_stamp: "Formatted at 2026-09-23 20:17:33 on dist-test-slave-vl24"
server_key: "970ed5e16c4437a69afe1c99e6dddbe5"
server_key_iv: "3f87370d23b9e0384a4d38ef3141e8be"
server_key_version: "kuduclusterkey@0"
I20260923 20:17:33.541290 16862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "ee2583c7bb984b5890b280896a01dfe9"
format_stamp: "Formatted at 2026-09-23 20:17:33 on dist-test-slave-vl24"
server_key: "970ed5e16c4437a69afe1c99e6dddbe5"
server_key_iv: "3f87370d23b9e0384a4d38ef3141e8be"
server_key_version: "kuduclusterkey@0"
I20260923 20:17:33.548926 16862 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260923 20:17:33.586282 16882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:17:33.588629 16862 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.006s
I20260923 20:17:33.588919 16862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "ee2583c7bb984b5890b280896a01dfe9"
format_stamp: "Formatted at 2026-09-23 20:17:33 on dist-test-slave-vl24"
server_key: "970ed5e16c4437a69afe1c99e6dddbe5"
server_key_iv: "3f87370d23b9e0384a4d38ef3141e8be"
server_key_version: "kuduclusterkey@0"
I20260923 20:17:33.589304 16862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:17:33.612078 16862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:17:33.616319 16862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:17:33.616673 16862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:17:33.617803 16862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:17:33.619294 16862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:17:33.619369 16862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:17:33.619501 16862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:17:33.619565 16862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:17:33.691355 16862 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:40557
I20260923 20:17:33.691367 16994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:40557 every 8 connection(s)
I20260923 20:17:33.693027 16862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:17:33.695835 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 16862
I20260923 20:17:33.696095 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:17:33.697139 5453 external_mini_cluster.cc:1428] Setting key bd24ffcb466e1d8cb0d436b3ccf7f1cf
Sep 23 20:17:33 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194653, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
I20260923 20:17:33.740410 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:33.702961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46359 (local address 127.5.83.65:57649)
0923 20:17:33.715551 (+ 12590us) negotiation.cc:107] Waiting for socket to connect
0923 20:17:33.715602 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:17:33.717789 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:17:33.720076 (+ 2287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:17:33.720084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:17:33.720638 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:17:33.721456 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:33.721482 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:33.723013 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:33.723019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:33.725171 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:33.725198 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:33.726478 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:33.726485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:33.726827 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:33.727693 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:17:33.727725 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:17:33.730931 (+ 3206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:17:33.735167 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:17:33.735178 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:17:33.735198 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:17:33.735823 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:17:33.736264 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:17:33.736269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:17:33.736273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:17:33.736551 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:17:33.737465 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:17:33.737473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:17:33.737968 (+ 495us) client_negotiation.cc:769] Sending connection context
0923 20:17:33.738317 (+ 349us) client_negotiation.cc:240] Negotiation successful
0923 20:17:33.738928 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12235,"thread_start_us":15140,"threads_started":1}
I20260923 20:17:33.741557 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:33.702300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:57649 (local address 127.5.83.126:46359)
0923 20:17:33.719346 (+ 17046us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:33.719366 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:33.719388 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:17:33.719451 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:33.719457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:33.719564 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:33.719678 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:17:33.721674 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:33.722808 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:33.725388 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:33.726040 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:33.726162 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:33.731215 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:17:33.731222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:17:33.731228 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:17:33.731281 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:17:33.734977 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:33.735975 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:33.735981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:33.735985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:33.736142 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:33.736745 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:33.736751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:33.736755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:33.737215 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:17:33.737375 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:33.740861 (+ 3486us) server_negotiation.cc:299] Negotiation successful
0923 20:17:33.741148 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16911,"thread_start_us":18292,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
I20260923 20:17:33.746675 16995 heartbeater.cc:347] Connected to a master server at 127.5.83.126:46359
I20260923 20:17:33.747031 16995 heartbeater.cc:464] Registering TS with master...
I20260923 20:17:33.747952 16995 heartbeater.cc:511] Master 127.5.83.126:46359 requested a full tablet report, sending...
I20260923 20:17:33.750717 16782 ts_manager.cc:194] Registered new tserver with Master: ee2583c7bb984b5890b280896a01dfe9 (127.5.83.65:40557)
I20260923 20:17:33.752822 16782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:57649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:17:33.780035 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.84:40489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46359
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:46303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:17:33 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194653, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
Sep 23 20:17:34 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194653, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
W20260923 20:17:34.232579 17001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:17:34.232986 17001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:17:34.233188 17001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:17:34.243660 17001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:17:34.243831 17001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:17:34.244038 17001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:17:34.259075 17001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:46303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.83.84:40489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:17:34.261229 17001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:17:34.263330 17001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:17:34.339628 17011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:17:34.340358 17008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:17:34.347873 17009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:17:34.360478 17001 server_base.cc:1034] running on GCE node
I20260923 20:17:34.361559 17001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:17:34.362918 17001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:17:34.364161 17001 hybrid_clock.cc:648] HybridClock initialized: now 1790194654364036 us; error 118 us; skew 500 ppm
I20260923 20:17:34.364634 17001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:17:34 dist-test-slave-vl24 krb5kdc[16306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194654, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:17:34.372043 17001 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:17:34.392918 17001 webserver.cc:460] Webserver started at http://127.5.83.66:44855/ using document root <none> and password file <none>
I20260923 20:17:34.394266 17001 fs_manager.cc:359] Metadata directory not provided
I20260923 20:17:34.394471 17001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:17:34.394979 17001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:17:34 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194654, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:17:34.417456 17001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "3ce779b5ebae45a79ee758d481ee70ef"
format_stamp: "Formatted at 2026-09-23 20:17:34 on dist-test-slave-vl24"
server_key: "34af87cb0709eab336a921768a69cf30"
server_key_iv: "500d0b01887279891dff6c7359d0cb34"
server_key_version: "kuduclusterkey@0"
I20260923 20:17:34.418608 17001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "3ce779b5ebae45a79ee758d481ee70ef"
format_stamp: "Formatted at 2026-09-23 20:17:34 on dist-test-slave-vl24"
server_key: "34af87cb0709eab336a921768a69cf30"
server_key_iv: "500d0b01887279891dff6c7359d0cb34"
server_key_version: "kuduclusterkey@0"
I20260923 20:17:34.426345 17001 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260923 20:17:34.466425 17018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:17:34.468711 17001 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.005s sys 0.016s
I20260923 20:17:34.469010 17001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "3ce779b5ebae45a79ee758d481ee70ef"
format_stamp: "Formatted at 2026-09-23 20:17:34 on dist-test-slave-vl24"
server_key: "34af87cb0709eab336a921768a69cf30"
server_key_iv: "500d0b01887279891dff6c7359d0cb34"
server_key_version: "kuduclusterkey@0"
I20260923 20:17:34.469383 17001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:17:34.520364 17001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:17:34.526206 17001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:17:34.526530 17001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:17:34.527778 17001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:17:34.530094 17001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:17:34.530189 17001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:17:34.530323 17001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:17:34.530391 17001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:17:34.597486 17001 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:42037
I20260923 20:17:34.597577 17137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:42037 every 8 connection(s)
I20260923 20:17:34.598905 17001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:17:34.606074 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 17001
I20260923 20:17:34.606262 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:17:34.606797 5453 external_mini_cluster.cc:1428] Setting key 1e85ade12d23c0991c830b5ca043e51a
Sep 23 20:17:34 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194654, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:17:34.624118 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:34.602034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:40293 (local address 127.5.83.126:46359)
0923 20:17:34.602538 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:34.602542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:34.603727 (+ 1185us) server_negotiation.cc:407] Connection header received
0923 20:17:34.605839 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:34.605847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:34.605954 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:34.606092 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:17:34.608003 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:34.609273 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:34.611918 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:34.612655 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:34.612802 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:34.617684 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:17:34.617692 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:17:34.617698 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:17:34.617744 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:17:34.620943 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:34.621774 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:34.621783 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:34.621788 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:34.621913 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:34.622343 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:34.622351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:34.622356 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:34.622648 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:17:34.622796 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:34.623516 (+ 720us) server_negotiation.cc:299] Negotiation successful
0923 20:17:34.623738 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":398,"thread_start_us":275,"threads_started":1}
I20260923 20:17:34.625272 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:34.602658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46359 (local address 127.5.83.66:40293)
0923 20:17:34.603560 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0923 20:17:34.603599 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:17:34.605496 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:17:34.606366 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:17:34.606378 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:17:34.607016 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:17:34.607789 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:34.607815 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:34.609467 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:34.609474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:34.611741 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:34.611758 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:34.612901 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:34.612911 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:34.613257 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:34.614196 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:17:34.614234 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:17:34.617368 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:17:34.621100 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:17:34.621110 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:17:34.621130 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:17:34.621637 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:17:34.622033 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:17:34.622038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:17:34.622042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:17:34.622234 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:17:34.622785 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:17:34.622794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:17:34.623136 (+ 342us) client_negotiation.cc:769] Sending connection context
0923 20:17:34.623413 (+ 277us) client_negotiation.cc:240] Negotiation successful
0923 20:17:34.623963 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":453,"threads_started":1}
I20260923 20:17:34.626822 17138 heartbeater.cc:347] Connected to a master server at 127.5.83.126:46359
I20260923 20:17:34.627151 17138 heartbeater.cc:464] Registering TS with master...
I20260923 20:17:34.627858 17138 heartbeater.cc:511] Master 127.5.83.126:46359 requested a full tablet report, sending...
I20260923 20:17:34.629490 16782 ts_manager.cc:194] Registered new tserver with Master: 3ce779b5ebae45a79ee758d481ee70ef (127.5.83.66:42037)
I20260923 20:17:34.630522 16782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:40293
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:17:34.672281 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.84:40489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46359
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:46303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:17:34.757776 16995 heartbeater.cc:503] Master 127.5.83.126:46359 was elected leader, sending a full tablet report...
W20260923 20:17:35.031418 17144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:17:35.031692 17144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:17:35.031888 17144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:17:35.039367 17144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:17:35.039518 17144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:17:35.039701 17144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:17:35.050736 17144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:46303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.83.84:40489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:17:35.052233 17144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:17:35.053639 17144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:17:35.066166 17149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:17:35.066334 17150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:17:35.067540 17152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:17:35.068712 17144 server_base.cc:1034] running on GCE node
I20260923 20:17:35.069278 17144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:17:35.070420 17144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:17:35.071641 17144 hybrid_clock.cc:648] HybridClock initialized: now 1790194655071598 us; error 43 us; skew 500 ppm
I20260923 20:17:35.071983 17144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:17:35 dist-test-slave-vl24 krb5kdc[16306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194655, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:17:35.075156 17144 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:17:35.076814 17144 webserver.cc:460] Webserver started at http://127.5.83.67:44721/ using document root <none> and password file <none>
I20260923 20:17:35.077616 17144 fs_manager.cc:359] Metadata directory not provided
I20260923 20:17:35.077718 17144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:17:35.077988 17144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:17:35 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194655, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:17:35.094385 17144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "15b16354275a44a3915fdd781dfdf39c"
format_stamp: "Formatted at 2026-09-23 20:17:35 on dist-test-slave-vl24"
server_key: "27985b3259e578ce9838634185c20d89"
server_key_iv: "f2f27f590bc1f1749d7431d49f59f9a1"
server_key_version: "kuduclusterkey@0"
I20260923 20:17:35.095513 17144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "15b16354275a44a3915fdd781dfdf39c"
format_stamp: "Formatted at 2026-09-23 20:17:35 on dist-test-slave-vl24"
server_key: "27985b3259e578ce9838634185c20d89"
server_key_iv: "f2f27f590bc1f1749d7431d49f59f9a1"
server_key_version: "kuduclusterkey@0"
I20260923 20:17:35.101989 17144 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260923 20:17:35.118810 17159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:17:35.120736 17144 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 20:17:35.120942 17144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "15b16354275a44a3915fdd781dfdf39c"
format_stamp: "Formatted at 2026-09-23 20:17:35 on dist-test-slave-vl24"
server_key: "27985b3259e578ce9838634185c20d89"
server_key_iv: "f2f27f590bc1f1749d7431d49f59f9a1"
server_key_version: "kuduclusterkey@0"
I20260923 20:17:35.121196 17144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:17:35.144618 17144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:17:35.148981 17144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:17:35.149322 17144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:17:35.150337 17144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:17:35.151889 17144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:17:35.151965 17144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:17:35.152053 17144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:17:35.152096 17144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:17:35.190821 17144 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:37323
I20260923 20:17:35.190869 17271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:37323 every 8 connection(s)
I20260923 20:17:35.192129 17144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:17:35.194913 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 17144
I20260923 20:17:35.195103 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:17:35.195770 5453 external_mini_cluster.cc:1428] Setting key 0db2711873cf52e4b212496bafe827a3
Sep 23 20:17:35 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194655, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:17:35.213876 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.194786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:32773 (local address 127.5.83.126:46359)
0923 20:17:35.195474 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:35.195480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:35.196845 (+ 1365us) server_negotiation.cc:407] Connection header received
0923 20:17:35.199135 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:35.199152 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:35.199256 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:35.199422 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:17:35.201327 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.202063 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.203754 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.204148 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.204237 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.207739 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:17:35.207744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:17:35.207747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:17:35.207781 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:17:35.210392 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:35.211299 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:35.211303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:35.211318 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:35.211510 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:35.212017 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:35.212023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:35.212026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:35.212476 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:17:35.212696 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:35.213399 (+ 703us) server_negotiation.cc:299] Negotiation successful
0923 20:17:35.213579 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":324,"threads_started":1}
I20260923 20:17:35.214843 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.195573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46359 (local address 127.5.83.67:32773)
0923 20:17:35.196656 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0923 20:17:35.196711 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 20:17:35.198804 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:17:35.199881 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:17:35.199895 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:17:35.200508 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:17:35.201158 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:35.201176 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.202216 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.202220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:35.203614 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:35.203637 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.204343 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.204348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:35.204644 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.205375 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:17:35.205406 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:17:35.207526 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:17:35.210531 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:17:35.210540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:17:35.210557 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:17:35.211152 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:17:35.211633 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:17:35.211638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:17:35.211641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:17:35.211892 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:17:35.212689 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:17:35.212695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:17:35.213029 (+ 334us) client_negotiation.cc:769] Sending connection context
0923 20:17:35.213256 (+ 227us) client_negotiation.cc:240] Negotiation successful
0923 20:17:35.213705 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":426,"threads_started":1}
I20260923 20:17:35.216359 17272 heartbeater.cc:347] Connected to a master server at 127.5.83.126:46359
I20260923 20:17:35.216707 17272 heartbeater.cc:464] Registering TS with master...
I20260923 20:17:35.217399 17272 heartbeater.cc:511] Master 127.5.83.126:46359 requested a full tablet report, sending...
I20260923 20:17:35.218787 16782 ts_manager.cc:194] Registered new tserver with Master: 15b16354275a44a3915fdd781dfdf39c (127.5.83.67:37323)
I20260923 20:17:35.219551 16782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:32773
I20260923 20:17:35.224512 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:17:35.241525 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.229059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.5.83.126:46359)
0923 20:17:35.229268 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:35.229278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:35.229647 (+ 369us) server_negotiation.cc:407] Connection header received
0923 20:17:35.229780 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:35.229784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:35.229863 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:35.229944 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:17:35.231508 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.232272 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.233775 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.234283 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.234374 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.236833 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:17:35.236838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:17:35.236841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:17:35.236873 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:17:35.239255 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:35.239902 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:35.239906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:35.239908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:35.240027 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:35.240331 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:35.240334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:35.240336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:35.240602 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:17:35.240780 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:35.241026 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 20:17:35.241186 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:17:35.245316 16782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:17:35.252580 16782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:17:35.274655 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.265124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.5.83.66:42037)
0923 20:17:35.267250 (+ 2126us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:35.267257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:35.267278 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:17:35.267355 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:35.267360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:35.267557 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:35.268248 (+ 691us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:17:35.268818 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.270438 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.272585 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.273492 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.273644 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.273884 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:35.274159 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 20:17:35.274326 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1979,"thread_start_us":1876,"threads_started":1}
I20260923 20:17:35.274796 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.264888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:42037 (local address 127.0.0.1:47680)
0923 20:17:35.265692 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0923 20:17:35.265739 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:17:35.265985 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:17:35.267779 (+ 1794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:17:35.267787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:17:35.267827 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:17:35.268395 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:35.268407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.270600 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.270606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:35.272393 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:35.272413 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.273697 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.273704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:35.273894 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.273948 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 20:17:35.274105 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 20:17:35.274311 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":24,"spinlock_wait_cycles":896,"thread_start_us":323,"threads_started":1}
I20260923 20:17:35.278368 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.265486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:40557 (local address 127.0.0.1:51908)
0923 20:17:35.266677 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0923 20:17:35.266693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:17:35.266808 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:17:35.267779 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:17:35.267787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:17:35.267833 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:17:35.268749 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:35.268763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.270643 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.270654 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:35.274687 (+ 4033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:35.274699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.277529 (+ 2830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.277537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:35.277710 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.277742 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:17:35.277843 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 20:17:35.277980 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"mutex_wait_us":198,"spinlock_wait_cycles":8192,"thread_start_us":960,"threads_started":1}
I20260923 20:17:35.278725 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.266373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51908 (local address 127.5.83.65:40557)
0923 20:17:35.267162 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:35.267170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:35.267197 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:17:35.267299 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:35.267308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:35.267544 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:35.268499 (+ 955us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:17:35.268945 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.270519 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.275598 (+ 5079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.277211 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.277340 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.277589 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:35.277907 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 20:17:35.278093 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":725,"threads_started":1}
I20260923 20:17:35.279295 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.267006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:37323 (local address 127.0.0.1:47508)
0923 20:17:35.267727 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0923 20:17:35.267744 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:17:35.267868 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:17:35.269139 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:17:35.269145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:17:35.269167 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:17:35.269523 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:35.269532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.270887 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.270894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:35.275954 (+ 5060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:35.275967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.278431 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.278437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:35.278611 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.278647 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 20:17:35.278729 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 20:17:35.278873 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":134,"thread_start_us":847,"threads_started":1}
I20260923 20:17:35.279048 17073 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed49a77b205472b893284627f1f1079 (DEFAULT_TABLE table=test-table [id=bbf1ca183e23423191678e0852cb708e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:17:35.280280 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.267592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.5.83.67:37323)
0923 20:17:35.268569 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:35.268579 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:35.268614 (+ 35us) server_negotiation.cc:407] Connection header received
0923 20:17:35.268704 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:35.268712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:35.268934 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:35.269130 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:17:35.269683 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.270740 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.276130 (+ 5390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.277958 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.278076 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.278284 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:35.279874 (+ 1590us) server_negotiation.cc:299] Negotiation successful
0923 20:17:35.280034 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":1275,"threads_started":1}
I20260923 20:17:35.280740 17073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed49a77b205472b893284627f1f1079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:17:35.282717 16930 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed49a77b205472b893284627f1f1079 (DEFAULT_TABLE table=test-table [id=bbf1ca183e23423191678e0852cb708e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:17:35.283181 17207 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed49a77b205472b893284627f1f1079 (DEFAULT_TABLE table=test-table [id=bbf1ca183e23423191678e0852cb708e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:17:35.284574 17207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed49a77b205472b893284627f1f1079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:17:35.284584 16930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed49a77b205472b893284627f1f1079. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:17:35.295451 17291 tablet_bootstrap.cc:492] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c: Bootstrap starting.
I20260923 20:17:35.298525 17292 tablet_bootstrap.cc:492] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef: Bootstrap starting.
I20260923 20:17:35.299608 17293 tablet_bootstrap.cc:492] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9: Bootstrap starting.
I20260923 20:17:35.300060 17291 tablet_bootstrap.cc:654] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c: Neither blocks nor log segments found. Creating new log.
I20260923 20:17:35.301478 17291 log.cc:824] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:17:35.302656 17292 tablet_bootstrap.cc:654] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef: Neither blocks nor log segments found. Creating new log.
I20260923 20:17:35.303113 17293 tablet_bootstrap.cc:654] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9: Neither blocks nor log segments found. Creating new log.
I20260923 20:17:35.303890 17292 log.cc:824] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef: Log is configured to *not* fsync() on all Append() calls
I20260923 20:17:35.304114 17293 log.cc:824] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:17:35.304950 17291 tablet_bootstrap.cc:492] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c: No bootstrap required, opened a new log
I20260923 20:17:35.305275 17291 ts_tablet_manager.cc:1397] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260923 20:17:35.307794 17292 tablet_bootstrap.cc:492] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef: No bootstrap required, opened a new log
I20260923 20:17:35.308064 17292 ts_tablet_manager.cc:1397] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260923 20:17:35.316128 17291 raft_consensus.cc:348] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce779b5ebae45a79ee758d481ee70ef" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42037 } } peers { permanent_uuid: "15b16354275a44a3915fdd781dfdf39c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 37323 } } peers { permanent_uuid: "ee2583c7bb984b5890b280896a01dfe9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40557 } }
I20260923 20:17:35.317569 17291 raft_consensus.cc:374] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:17:35.317742 17291 raft_consensus.cc:729] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b16354275a44a3915fdd781dfdf39c, State: Initialized, Role: FOLLOWER
I20260923 20:17:35.318634 17291 consensus_queue.cc:260] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce779b5ebae45a79ee758d481ee70ef" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42037 } } peers { permanent_uuid: "15b16354275a44a3915fdd781dfdf39c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 37323 } } peers { permanent_uuid: "ee2583c7bb984b5890b280896a01dfe9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40557 } }
I20260923 20:17:35.318593 17292 raft_consensus.cc:348] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce779b5ebae45a79ee758d481ee70ef" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42037 } } peers { permanent_uuid: "15b16354275a44a3915fdd781dfdf39c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 37323 } } peers { permanent_uuid: "ee2583c7bb984b5890b280896a01dfe9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40557 } }
I20260923 20:17:35.319069 17292 raft_consensus.cc:374] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:17:35.319227 17292 raft_consensus.cc:729] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce779b5ebae45a79ee758d481ee70ef, State: Initialized, Role: FOLLOWER
I20260923 20:17:35.319794 17272 heartbeater.cc:503] Master 127.5.83.126:46359 was elected leader, sending a full tablet report...
I20260923 20:17:35.320123 17292 consensus_queue.cc:260] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce779b5ebae45a79ee758d481ee70ef" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42037 } } peers { permanent_uuid: "15b16354275a44a3915fdd781dfdf39c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 37323 } } peers { permanent_uuid: "ee2583c7bb984b5890b280896a01dfe9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40557 } }
I20260923 20:17:35.320672 17293 tablet_bootstrap.cc:492] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9: No bootstrap required, opened a new log
I20260923 20:17:35.320695 17291 ts_tablet_manager.cc:1428] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260923 20:17:35.321060 17293 ts_tablet_manager.cc:1397] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.016s
I20260923 20:17:35.321681 17138 heartbeater.cc:503] Master 127.5.83.126:46359 was elected leader, sending a full tablet report...
I20260923 20:17:35.322257 17292 ts_tablet_manager.cc:1428] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260923 20:17:35.330149 17293 raft_consensus.cc:348] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce779b5ebae45a79ee758d481ee70ef" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42037 } } peers { permanent_uuid: "15b16354275a44a3915fdd781dfdf39c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 37323 } } peers { permanent_uuid: "ee2583c7bb984b5890b280896a01dfe9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40557 } }
I20260923 20:17:35.330708 17293 raft_consensus.cc:374] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:17:35.330845 17293 raft_consensus.cc:729] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2583c7bb984b5890b280896a01dfe9, State: Initialized, Role: FOLLOWER
I20260923 20:17:35.331488 17293 consensus_queue.cc:260] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce779b5ebae45a79ee758d481ee70ef" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42037 } } peers { permanent_uuid: "15b16354275a44a3915fdd781dfdf39c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 37323 } } peers { permanent_uuid: "ee2583c7bb984b5890b280896a01dfe9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40557 } }
I20260923 20:17:35.333216 17293 ts_tablet_manager.cc:1428] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
W20260923 20:17:35.351241 17139 tablet.cc:2369] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:17:35.443926 17273 tablet.cc:2369] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:17:35.465919 16998 tablet.cc:2369] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:17:35.600621 17299 raft_consensus.cc:482] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:17:35.600950 17299 raft_consensus.cc:504] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce779b5ebae45a79ee758d481ee70ef" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42037 } } peers { permanent_uuid: "15b16354275a44a3915fdd781dfdf39c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 37323 } } peers { permanent_uuid: "ee2583c7bb984b5890b280896a01dfe9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40557 } }
I20260923 20:17:35.602923 17299 leader_election.cc:288] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ce779b5ebae45a79ee758d481ee70ef (127.5.83.66:42037), 15b16354275a44a3915fdd781dfdf39c (127.5.83.67:37323)
I20260923 20:17:35.610740 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.603297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:50125 (local address 127.5.83.66:42037)
0923 20:17:35.603523 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:35.603529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:35.604176 (+ 647us) server_negotiation.cc:407] Connection header received
0923 20:17:35.604428 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:35.604438 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:35.604594 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:35.604678 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:17:35.605597 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.606717 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.608883 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.609723 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.609830 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.609898 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:35.610333 (+ 435us) server_negotiation.cc:299] Negotiation successful
0923 20:17:35.610489 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:17:35.610771 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.604500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:48023 (local address 127.5.83.67:37323)
0923 20:17:35.604974 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:35.604980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:35.605155 (+ 175us) server_negotiation.cc:407] Connection header received
0923 20:17:35.605333 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:35.605340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:35.605429 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:35.605528 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:17:35.606142 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.607081 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.609082 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.609901 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.610047 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.610146 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:35.610415 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 20:17:35.610550 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316}
I20260923 20:17:35.610832 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.604237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:37323 (local address 127.5.83.65:48023)
0923 20:17:35.605033 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0923 20:17:35.605050 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:17:35.605197 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:17:35.605567 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:17:35.605571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:17:35.605599 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:17:35.605955 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:35.605966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.607224 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.607229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:35.608906 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:35.608923 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.610112 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.610117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:35.610228 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.610274 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 20:17:35.610367 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 20:17:35.610486 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":476,"threads_started":1}
I20260923 20:17:35.610862 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.603289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:42037 (local address 127.5.83.65:50125)
0923 20:17:35.604056 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0923 20:17:35.604077 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:17:35.604222 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:17:35.604981 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:17:35.604986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:17:35.605020 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:17:35.605435 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:35.605447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.606848 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.606853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:35.608729 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:17:35.608740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.609947 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.609956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:17:35.610148 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.610189 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 20:17:35.610276 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 20:17:35.610411 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":7,"thread_start_us":460,"threads_started":1}
I20260923 20:17:35.612054 17227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed49a77b205472b893284627f1f1079" candidate_uuid: "ee2583c7bb984b5890b280896a01dfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15b16354275a44a3915fdd781dfdf39c" is_pre_election: true
I20260923 20:17:35.612057 17093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed49a77b205472b893284627f1f1079" candidate_uuid: "ee2583c7bb984b5890b280896a01dfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce779b5ebae45a79ee758d481ee70ef" is_pre_election: true
I20260923 20:17:35.612638 17227 raft_consensus.cc:2454] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee2583c7bb984b5890b280896a01dfe9 in term 0.
I20260923 20:17:35.612638 17093 raft_consensus.cc:2454] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee2583c7bb984b5890b280896a01dfe9 in term 0.
I20260923 20:17:35.613301 16883 leader_election.cc:302] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b16354275a44a3915fdd781dfdf39c, ee2583c7bb984b5890b280896a01dfe9; no voters:
I20260923 20:17:35.613662 17299 raft_consensus.cc:2781] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:17:35.613788 17299 raft_consensus.cc:482] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:17:35.613920 17299 raft_consensus.cc:3037] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:17:35.616175 17299 raft_consensus.cc:504] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce779b5ebae45a79ee758d481ee70ef" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42037 } } peers { permanent_uuid: "15b16354275a44a3915fdd781dfdf39c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 37323 } } peers { permanent_uuid: "ee2583c7bb984b5890b280896a01dfe9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40557 } }
I20260923 20:17:35.616796 17299 leader_election.cc:288] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [CANDIDATE]: Term 1 election: Requested vote from peers 3ce779b5ebae45a79ee758d481ee70ef (127.5.83.66:42037), 15b16354275a44a3915fdd781dfdf39c (127.5.83.67:37323)
I20260923 20:17:35.617313 17093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed49a77b205472b893284627f1f1079" candidate_uuid: "ee2583c7bb984b5890b280896a01dfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce779b5ebae45a79ee758d481ee70ef"
I20260923 20:17:35.617419 17227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed49a77b205472b893284627f1f1079" candidate_uuid: "ee2583c7bb984b5890b280896a01dfe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15b16354275a44a3915fdd781dfdf39c"
I20260923 20:17:35.617592 17093 raft_consensus.cc:3037] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:17:35.617651 17227 raft_consensus.cc:3037] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:17:35.620870 17093 raft_consensus.cc:2454] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee2583c7bb984b5890b280896a01dfe9 in term 1.
I20260923 20:17:35.620870 17227 raft_consensus.cc:2454] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee2583c7bb984b5890b280896a01dfe9 in term 1.
I20260923 20:17:35.621276 16883 leader_election.cc:302] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ce779b5ebae45a79ee758d481ee70ef, ee2583c7bb984b5890b280896a01dfe9; no voters:
I20260923 20:17:35.621557 17299 raft_consensus.cc:2781] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:17:35.622004 17299 raft_consensus.cc:686] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 1 LEADER]: Becoming Leader. State: Replica: ee2583c7bb984b5890b280896a01dfe9, State: Running, Role: LEADER
I20260923 20:17:35.622593 17299 consensus_queue.cc:237] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce779b5ebae45a79ee758d481ee70ef" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42037 } } peers { permanent_uuid: "15b16354275a44a3915fdd781dfdf39c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 37323 } } peers { permanent_uuid: "ee2583c7bb984b5890b280896a01dfe9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40557 } }
I20260923 20:17:35.627384 16782 catalog_manager.cc:5495] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee2583c7bb984b5890b280896a01dfe9 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "ee2583c7bb984b5890b280896a01dfe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce779b5ebae45a79ee758d481ee70ef" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15b16354275a44a3915fdd781dfdf39c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 37323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee2583c7bb984b5890b280896a01dfe9" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 40557 } health_report { overall_health: HEALTHY } } }
I20260923 20:17:35.643923 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.638940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.5.83.65:40557)
0923 20:17:35.639137 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:35.639142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:35.639158 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:17:35.639242 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:35.639245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:35.639318 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:35.639447 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:17:35.639754 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.640496 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.641897 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.642282 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.642373 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.642828 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:17:35.643360 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:17:35.643474 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:35.643550 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 20:17:35.643615 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:17:35.652786 17093 raft_consensus.cc:1261] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef [term 1 FOLLOWER]: Refusing update from remote peer ee2583c7bb984b5890b280896a01dfe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:17:35.652786 17227 raft_consensus.cc:1261] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c [term 1 FOLLOWER]: Refusing update from remote peer ee2583c7bb984b5890b280896a01dfe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:17:35.653831 17302 consensus_queue.cc:1035] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15b16354275a44a3915fdd781dfdf39c" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:17:35.654326 17299 consensus_queue.cc:1035] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ce779b5ebae45a79ee758d481ee70ef" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:17:35.667933 17306 mvcc.cc:205] Tried to move back new op lower bound from 7332637309540712448 to 7332637309434515456. Current Snapshot: MvccSnapshot[applied={T|T < 7332637309540712448}]
I20260923 20:17:35.683177 16782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:17:35.683475 16782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:17:35.688191 16782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f644863bfb7347cfbf09e363c4c84093: Sending DeleteTablet for 3 replicas of tablet 1ed49a77b205472b893284627f1f1079
I20260923 20:17:35.689873 17073 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed49a77b205472b893284627f1f1079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:17:35 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:47680
I20260923 20:17:35.689885 16930 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed49a77b205472b893284627f1f1079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:17:35 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:51908
I20260923 20:17:35.690167 17207 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed49a77b205472b893284627f1f1079 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:17:35 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:47508
I20260923 20:17:35.694260 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 16862
I20260923 20:17:35.713804 17316 tablet_replica.cc:330] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c: stopping tablet replica
I20260923 20:17:35.714493 17316 raft_consensus.cc:2229] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:17:35.715157 17316 raft_consensus.cc:2258] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:17:35.721514 17315 tablet_replica.cc:330] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9: stopping tablet replica
I20260923 20:17:35.722134 17315 raft_consensus.cc:2229] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:17:35.723008 17315 raft_consensus.cc:2258] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:17:35.723953 17314 tablet_replica.cc:330] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef: stopping tablet replica
I20260923 20:17:35.724323 17314 raft_consensus.cc:2229] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:17:35.727777 17316 ts_tablet_manager.cc:1905] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:17:35.731945 17315 ts_tablet_manager.cc:1905] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:17:35.725345 17314 raft_consensus.cc:2258] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:17:35.735932 17316 ts_tablet_manager.cc:1918] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:17:35.736116 17316 log.cc:1196] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/1ed49a77b205472b893284627f1f1079
I20260923 20:17:35.736725 17316 ts_tablet_manager.cc:1939] T 1ed49a77b205472b893284627f1f1079 P 15b16354275a44a3915fdd781dfdf39c: Deleting consensus metadata
I20260923 20:17:35.738523 17314 ts_tablet_manager.cc:1905] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:17:35.739395 16766 catalog_manager.cc:4841] TS 15b16354275a44a3915fdd781dfdf39c (127.5.83.67:37323): tablet 1ed49a77b205472b893284627f1f1079 (table test-table [id=bbf1ca183e23423191678e0852cb708e]) successfully deleted
I20260923 20:17:35.744923 17315 ts_tablet_manager.cc:1918] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:17:35.745116 17315 log.cc:1196] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/1ed49a77b205472b893284627f1f1079
I20260923 20:17:35.745479 17314 ts_tablet_manager.cc:1918] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:17:35.745657 17314 log.cc:1196] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/1ed49a77b205472b893284627f1f1079
I20260923 20:17:35.745705 17315 ts_tablet_manager.cc:1939] T 1ed49a77b205472b893284627f1f1079 P ee2583c7bb984b5890b280896a01dfe9: Deleting consensus metadata
I20260923 20:17:35.746181 17314 ts_tablet_manager.cc:1939] T 1ed49a77b205472b893284627f1f1079 P 3ce779b5ebae45a79ee758d481ee70ef: Deleting consensus metadata
Sep 23 20:17:35 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:17:35.747660 16769 catalog_manager.cc:4841] TS ee2583c7bb984b5890b280896a01dfe9 (127.5.83.65:40557): tablet 1ed49a77b205472b893284627f1f1079 (table test-table [id=bbf1ca183e23423191678e0852cb708e]) successfully deleted
I20260923 20:17:35.747977 16766 catalog_manager.cc:4841] TS 3ce779b5ebae45a79ee758d481ee70ef (127.5.83.66:42037): tablet 1ed49a77b205472b893284627f1f1079 (table test-table [id=bbf1ca183e23423191678e0852cb708e]) successfully deleted
I20260923 20:17:35.754683 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.695535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51928 (local address 127.5.83.65:40557)
0923 20:17:35.695726 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:35.695732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:35.725446 (+ 29714us) server_negotiation.cc:407] Connection header received
0923 20:17:35.725655 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:35.725661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:35.725752 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:35.725840 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:17:35.728667 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.730006 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.740751 (+ 10745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.741478 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.741650 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.747797 (+ 6147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:17:35.747814 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:17:35.747831 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:17:35.747888 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:17:35.751552 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:35.752356 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:35.752363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:35.752365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:35.752658 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:35.753070 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:35.753074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:35.753077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:35.753454 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:17:35.753767 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:35.754109 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 20:17:35.754349 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:17:35.755367 16990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:17:35.859218 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 16862
I20260923 20:17:35.879238 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 17001
Sep 23 20:17:35 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:17:35.894817 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:35.879968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.5.83.66:42037)
0923 20:17:35.880170 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:35.880176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:35.880296 (+ 120us) server_negotiation.cc:407] Connection header received
0923 20:17:35.880502 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:35.880510 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:35.880642 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:35.880747 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:17:35.881981 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.882660 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.884169 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:35.884674 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:35.884810 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:35.889137 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:17:35.889152 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:17:35.889169 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:17:35.889221 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:17:35.891858 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:35.892573 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:35.892585 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:35.892591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:35.892754 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:35.893194 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:35.893200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:35.893204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:35.893587 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:17:35.893879 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:35.894217 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 20:17:35.894447 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:17:35.895457 17133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:17:35.985405 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 17001
I20260923 20:17:36.001991 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 17144
Sep 23 20:17:36 dist-test-slave-vl24 krb5kdc[16306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:17:36.017441 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:17:36.002784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.5.83.67:37323)
0923 20:17:36.002977 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:17:36.002983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:17:36.003003 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:17:36.003094 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:17:36.003098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:17:36.003162 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:17:36.003240 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:17:36.004494 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:36.005248 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:36.006547 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:17:36.006935 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:17:36.007021 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:17:36.011657 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:17:36.011673 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:17:36.011689 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:17:36.011725 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:17:36.014703 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:36.015430 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:36.015437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:36.015440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:36.015575 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:17:36.015953 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:17:36.015957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:17:36.015959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:17:36.016358 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:17:36.016777 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0923 20:17:36.016907 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 20:17:36.017070 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:17:36.018123 17267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:17:36.107014 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 17144
I20260923 20:17:36.123742 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 16750
I20260923 20:17:36.125025 16812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:17:36.236865 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 16750
I20260923 20:17:36.259085 5453 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 20:17:36.595331 5453 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 20:17:36.595554 5453 mini_ranger.cc:66] Stopping Ranger...
I20260923 20:17:36.941359 5453 mini_ranger.cc:68] Stopped Ranger
2026-09-23 20:17:36.941 UTC [16336] LOG: received smart shutdown request
2026-09-23 20:17:36.945 UTC [16336] LOG: background worker "logical replication launcher" (PID 16345) exited with exit code 1
2026-09-23 20:17:36.945 UTC [16340] LOG: shutting down
2026-09-23 20:17:37.124 UTC [16336] LOG: database system is shut down
2026-09-23T20:17:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50302 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:17:37 dist-test-slave-vl24 krb5kdc[17330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:17:37 dist-test-slave-vl24 krb5kdc[17330](info): set up 2 sockets
Sep 23 20:17:37 dist-test-slave-vl24 krb5kdc[17330](info): commencing operation
krb5kdc: starting...
W20260923 20:17:38.386459 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:17:38 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:17:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:17:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/rangeradmin_127.5.83.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/rangeradmin_127.5.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/rangerlookup_127.5.83.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/rangerlookup_127.5.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/HTTP_127.5.83.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/HTTP_127.5.83.84@KRBTEST.COM.keytab.
I20260923 20:17:38.725859 5453 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCNgSr1/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/postgres -l logfile start
2026-09-23 20:17:41.263 UTC [17360] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 20:17:41.263 UTC [17360] LOG: listening on IPv4 address "127.5.83.84", port 59979
2026-09-23 20:17:41.267 UTC [17360] LOG: listening on Unix socket "/tmp/.s.PGSQL.59979"
2026-09-23 20:17:41.289 UTC [17363] LOG: database system was shut down at 2026-09-23 20:17:39 UTC
2026-09-23 20:17:41.293 UTC [17360] LOG: database system is ready to accept connections
I20260923 20:17:42.309821 5453 mini_postgres.cc:96] Postgres bound to 59979
2026-09-23 20:17:42.328 UTC [17371] FATAL: database "slave" does not exist
127.5.83.84:59979 - accepting connections
I20260923 20:17:42.329583 5453 mini_ranger.cc:161] Starting Ranger...
I20260923 20:17:42.357375 5453 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 20:17:42.851822 5453 mini_ranger.cc:87] Created ranger Postgres database
I20260923 20:17:42.851969 5453 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-admin
2026-09-23 20:17:43,152 [I] DB FLAVOR :POSTGRES
2026-09-23 20:17:43,153 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:17:43,153 [I] Checking connection..
2026-09-23 20:17:43,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 20:17:43,391 [I] Checking connection passed.
2026-09-23 20:17:43,392 [I] --------- Verifying version history table ---------
2026-09-23 20:17:43,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:17:43,636 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:17:43,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:17:43,869 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 20:17:43,870 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 20:17:43,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 20:17:44.122 UTC [17435] WARNING: there is no transaction in progress
2026-09-23 20:17:44,128 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 20:17:44,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 20:17:44,404 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 20:17:44,404 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:17:44,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 20:17:44,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 20:17:44,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vl24.c.gcp-upstream.internal','N') ;"
2026-09-23 20:17:45,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:17:45,411 [I] Table x_portal_user does not exist in database ranger
2026-09-23 20:17:45,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:17:45,662 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 20:17:45,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:17:45,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:17:46,150 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 20:17:46,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 20:17:47.128 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.146 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.167 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.184 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.200 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.220 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.281 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.288 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.302 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.312 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.323 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.338 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.348 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.355 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.361 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:47.367 UTC [17543] WARNING: there is no transaction in progress
2026-09-23 20:17:48,018 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 20:17:48,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 20:17:48,270 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 20:17:48,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 20:17:48,518 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 20:17:48,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 20:17:48,772 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 20:17:48,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 20:17:49,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 20:17:49,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vl24.c.gcp-upstream.internal';"
2026-09-23 20:17:49,502 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 20:17:49,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 20:17:49,761 [I] DB_PATCHES have already been applied
I20260923 20:17:49.767335 5453 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCNgSr1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 20:17:49.767424 5453 mini_ranger.cc:193] Using host: 127.5.83.84
I20260923 20:17:49.783413 5453 mini_ranger.cc:227] Ranger admin URL: http://127.5.83.84:60495
Sep 23, 2026 8:17:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:17:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:17:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 8:17:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 8:17:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 8:17:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 8:17:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 20:17:51.006927 5453 mini_ranger.cc:160] Time spent starting Ranger: real 8.677s user 0.000s sys 0.045s
Sep 23, 2026 8:18:10 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 8:18:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 23, 2026 8:18:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 8:18:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 8:18:10 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 8:18:11 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 20:18:12.191313 5453 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.83.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.83.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/rangerkms_127.5.83.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.83.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/rangerkms_127.5.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.83.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.83.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.83.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/HTTP_127.5.83.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.83.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/HTTP_127.5.83.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 20:18:12 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194692, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 20:18:12.368106 5453 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 20:18:12.390021 5453 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 20:18:13.277585 5453 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 20:18:13.277681 5453 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-kms
I20260923 20:18:13.277712 5453 mini_ranger_kms.cc:227] Using postgres at 127.5.83.84:59979
2026-09-23 20:18:13,643 [I] DB FLAVOR :POSTGRES
2026-09-23 20:18:13,643 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 20:18:13,643 [I] Checking connection
2026-09-23 20:18:13,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 20:18:13,874 [I] connection success
2026-09-23 20:18:13,874 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 20:18:13,874 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 20:18:13,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 20:18:14,108 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 20:18:14,109 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 20:18:14,109 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 20:18:14,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.83.84:59979/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 20:18:14,364 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 20:18:14.847360 5453 mini_ranger_kms.cc:310] Created kms service
I20260923 20:18:15.053040 5453 mini_ranger_kms.cc:326] Created kudu user
I20260923 20:18:15.102105 5453 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 20:18:15 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194692, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:18:15.502699 5453 mini_ranger_kms.cc:384] Added ranger policy
I20260923 20:18:15.503086 5453 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCNgSr1/build/asan/bin/postgresql.jar:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCNgSr1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 20:18:15.503158 5453 mini_ranger_kms.cc:240] Using host: 127.5.83.84
I20260923 20:18:15.520494 5453 mini_ranger_kms.cc:276] Ranger KMS PID: 17723
I20260923 20:18:15.520653 5453 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.83.84:49651
Sep 23, 2026 8:18:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 8:18:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 8:18:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 8:18:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 8:18:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 8:18:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 20:18:17.805663 5453 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.438s user 0.054s sys 0.329s
I20260923 20:18:17.805932 5453 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 20:18:17.806120 5453 mini_ranger_kms.cc:401] 127.5.83.84:49651/kms/v1/keys
Sep 23 20:18:17 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790194697, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.83.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 20:18:18 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194697, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.83.84@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
Sep 23 20:18:18 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194697, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.83.84@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
Sep 23 20:18:21 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:18:21.108995 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:44663
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:44663
--ranger_config_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.84:49651/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:18:21.460516 17774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:21.460830 17774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:21.460919 17774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:21.468196 17774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:18:21.468307 17774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:21.468353 17774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:21.468410 17774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:18:21.468457 17774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:18:21.478708 17774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.83.84:49651/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:44663
--ranger_config_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:21.479940 17774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:21.481240 17774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:21.493880 17779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:21.493913 17782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:21.493935 17780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:21.494652 17774 server_base.cc:1034] running on GCE node
I20260923 20:18:21.495479 17774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:21.497053 17774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:21.498286 17774 hybrid_clock.cc:648] HybridClock initialized: now 1790194701498231 us; error 43 us; skew 500 ppm
I20260923 20:18:21.498682 17774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:21 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194701, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:21.507076 17774 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:18:21.508617 17774 webserver.cc:460] Webserver started at http://127.5.83.126:46511/ using document root <none> and password file <none>
I20260923 20:18:21.509267 17774 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:21.509359 17774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:21.509608 17774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:18:21 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194701, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:18:21.559840 17774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "aae82e024955424ba5a05b688878d91c"
format_stamp: "Formatted at 2026-09-23 20:18:21 on dist-test-slave-vl24"
server_key: "5be2fd525b66b3ad6af34b7dbb9b1bd0"
server_key_iv: "82669ea49550ec4dfa6476b81cde4ff9"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:21.560848 17774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "aae82e024955424ba5a05b688878d91c"
format_stamp: "Formatted at 2026-09-23 20:18:21 on dist-test-slave-vl24"
server_key: "5be2fd525b66b3ad6af34b7dbb9b1bd0"
server_key_iv: "82669ea49550ec4dfa6476b81cde4ff9"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:21.566751 17774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:18:21.583628 17789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:21.585265 17774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:18:21.585458 17774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "aae82e024955424ba5a05b688878d91c"
format_stamp: "Formatted at 2026-09-23 20:18:21 on dist-test-slave-vl24"
server_key: "5be2fd525b66b3ad6af34b7dbb9b1bd0"
server_key_iv: "82669ea49550ec4dfa6476b81cde4ff9"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:21.585686 17774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:21.609457 17774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:21.613179 17774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:21.613507 17774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:21.632815 17774 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:44663
I20260923 20:18:21.632845 17840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:44663 every 8 connection(s)
I20260923 20:18:21.634253 17774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:18:21.635468 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 17774
I20260923 20:18:21.635703 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:18:21.636684 5453 external_mini_cluster.cc:1428] Setting key 71c8d778714c998740d9615791b131fa
I20260923 20:18:21.639225 17841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:18:21 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:21.652935 17841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c: Bootstrap starting.
I20260923 20:18:21.657276 17841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:21.658613 17841 log.cc:824] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:21.661124 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:21.638817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.5.83.126:44663)
0923 20:18:21.639614 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:21.639626 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:21.639666 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:18:21.641595 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:21.641617 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:21.642123 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:21.642471 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:21.644192 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:21.645608 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:21.647419 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:21.648067 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:21.648192 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:21.652896 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:21.652909 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:21.652935 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:21.652980 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:21.656750 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:21.657395 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:21.657403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:21.657411 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:21.657620 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:21.658009 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:21.658014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:21.658017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:21.658723 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:21.659003 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:21.659453 (+ 450us) server_negotiation.cc:299] Negotiation successful
0923 20:18:21.659795 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":394,"threads_started":1}
I20260923 20:18:21.661965 17841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c: No bootstrap required, opened a new log
I20260923 20:18:21.669492 17841 raft_consensus.cc:348] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae82e024955424ba5a05b688878d91c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 44663 } }
I20260923 20:18:21.669890 17841 raft_consensus.cc:374] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:21.669992 17841 raft_consensus.cc:729] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae82e024955424ba5a05b688878d91c, State: Initialized, Role: FOLLOWER
I20260923 20:18:21.670630 17841 consensus_queue.cc:260] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae82e024955424ba5a05b688878d91c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 44663 } }
I20260923 20:18:21.670856 17841 raft_consensus.cc:388] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:18:21.671053 17841 raft_consensus.cc:482] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:18:21.671219 17841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:21.673542 17841 raft_consensus.cc:504] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae82e024955424ba5a05b688878d91c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 44663 } }
I20260923 20:18:21.674054 17841 leader_election.cc:302] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aae82e024955424ba5a05b688878d91c; no voters:
I20260923 20:18:21.674630 17841 leader_election.cc:288] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:18:21.674793 17846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:18:21.675580 17846 raft_consensus.cc:686] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 1 LEADER]: Becoming Leader. State: Replica: aae82e024955424ba5a05b688878d91c, State: Running, Role: LEADER
I20260923 20:18:21.676134 17846 consensus_queue.cc:237] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae82e024955424ba5a05b688878d91c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 44663 } }
I20260923 20:18:21.676805 17841 sys_catalog.cc:564] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:18:21.679111 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aae82e024955424ba5a05b688878d91c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae82e024955424ba5a05b688878d91c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 44663 } } }
I20260923 20:18:21.679571 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:21.679575 17848 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aae82e024955424ba5a05b688878d91c. Latest consensus state: current_term: 1 leader_uuid: "aae82e024955424ba5a05b688878d91c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae82e024955424ba5a05b688878d91c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 44663 } } }
I20260923 20:18:21.679884 17848 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:21.681244 17855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:18:21.686398 17855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:18:21.687273 17841 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 20:18:21.704772 17855 catalog_manager.cc:1347] Generated new cluster ID: 3217dda913af4dd1bf46452d5121f923
I20260923 20:18:21.704942 17855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:18:21.726902 17855 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:18:21.729336 17855 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:18:21.744151 17855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c: Generated new TSK 0
I20260923 20:18:21.745402 17855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:18:22.492885 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.84:49651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:44663
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:18:22 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790194702, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 20:18:22.940477 17886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:22.940858 17886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:22.941015 17886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:22.948385 17886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:22.948535 17886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:22.948742 17886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:18:22.959008 17886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.83.84:49651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:44663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:22.960757 17886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:22.962460 17886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:22.980304 17897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:22.983655 17896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:22.987911 17899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:22.988654 17886 server_base.cc:1034] running on GCE node
I20260923 20:18:22.989486 17886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:22.990794 17886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:22.992126 17886 hybrid_clock.cc:648] HybridClock initialized: now 1790194702992054 us; error 58 us; skew 500 ppm
I20260923 20:18:22.992560 17886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:22 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194702, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:22.997063 17886 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:18:22.999306 17886 webserver.cc:460] Webserver started at http://127.5.83.65:34205/ using document root <none> and password file <none>
I20260923 20:18:23.000326 17886 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:23.000483 17886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:23.000931 17886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:18:23 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194702, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:18:23.024679 17886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "4bed110f85004da19bc137ba4c417751"
format_stamp: "Formatted at 2026-09-23 20:18:23 on dist-test-slave-vl24"
server_key: "eb665aa75e777f01f3da5f7759a5d629"
server_key_iv: "43f53f239166df53cb340716dc8bb8b0"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:23.026011 17886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "4bed110f85004da19bc137ba4c417751"
format_stamp: "Formatted at 2026-09-23 20:18:23 on dist-test-slave-vl24"
server_key: "eb665aa75e777f01f3da5f7759a5d629"
server_key_iv: "43f53f239166df53cb340716dc8bb8b0"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:23.033805 17886 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260923 20:18:23.054433 17906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:23.056610 17886 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:18:23.056915 17886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "4bed110f85004da19bc137ba4c417751"
format_stamp: "Formatted at 2026-09-23 20:18:23 on dist-test-slave-vl24"
server_key: "eb665aa75e777f01f3da5f7759a5d629"
server_key_iv: "43f53f239166df53cb340716dc8bb8b0"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:23.057296 17886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:23.094099 17886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:23.099543 17886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:23.099895 17886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:23.101052 17886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:23.102618 17886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:23.102722 17886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:23.102880 17886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:23.102954 17886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:23.180389 17886 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:44397
I20260923 20:18:23.182403 17886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:18:23.182737 18018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:44397 every 8 connection(s)
I20260923 20:18:23.188800 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 17886
I20260923 20:18:23.189075 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:18:23.190095 5453 external_mini_cluster.cc:1428] Setting key c14c708d745d552bd9f0755d738ffc03
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
Sep 23 20:18:23 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194702, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
I20260923 20:18:23.234017 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:23.186916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:44633 (local address 127.5.83.126:44663)
0923 20:18:23.209452 (+ 22536us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:23.209462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:23.209487 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:18:23.209560 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:23.209567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:23.209679 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:23.214969 (+ 5290us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:23.215191 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:23.216564 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:23.219058 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:23.219660 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:23.219814 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:23.225981 (+ 6167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:23.225986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:23.225989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:23.226025 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:23.229627 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:23.230723 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:23.230731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:23.230738 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:23.230896 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:23.231503 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:23.231511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:23.231517 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:23.231912 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:23.232054 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:23.233258 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0923 20:18:23.233537 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22360,"thread_start_us":22867,"threads_started":1}
I20260923 20:18:23.236896 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:23.187720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:44663 (local address 127.5.83.65:44633)
0923 20:18:23.204140 (+ 16420us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:23.204214 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:23.207399 (+ 3185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:23.210259 (+ 2860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:23.210275 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:23.211069 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:23.214393 (+ 3324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:23.214439 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:23.216774 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:23.216781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:23.218827 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:23.218846 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:23.219932 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:23.219940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:23.220289 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:23.221222 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:23.221263 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:23.225690 (+ 4427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:23.229828 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:23.229842 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:23.229866 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:23.230557 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:23.231031 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:23.231038 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:23.231044 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:23.231350 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:23.232140 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:23.232152 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:23.232684 (+ 532us) client_negotiation.cc:769] Sending connection context
0923 20:18:23.233010 (+ 326us) client_negotiation.cc:240] Negotiation successful
0923 20:18:23.235236 (+ 2226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16065,"thread_start_us":15745,"threads_started":1}
I20260923 20:18:23.239132 18019 heartbeater.cc:347] Connected to a master server at 127.5.83.126:44663
I20260923 20:18:23.239472 18019 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:23.240365 18019 heartbeater.cc:511] Master 127.5.83.126:44663 requested a full tablet report, sending...
I20260923 20:18:23.242923 17806 ts_manager.cc:194] Registered new tserver with Master: 4bed110f85004da19bc137ba4c417751 (127.5.83.65:44397)
I20260923 20:18:23.245348 17806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:44633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:18:23.270350 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.84:49651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:44663
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:18:23 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194702, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
Sep 23 20:18:23 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194702, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
W20260923 20:18:23.734614 18025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:23.735153 18025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:23.735383 18025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:23.747148 18025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:23.747411 18025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:23.747655 18025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:18:23.764499 18025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.83.84:49651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:44663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:23.766515 18025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:23.768268 18025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:23.802230 18033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:23.808513 18036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:23.808962 18025 server_base.cc:1034] running on GCE node
W20260923 20:18:23.812834 18032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:23.813825 18025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:23.816468 18025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:23.817811 18025 hybrid_clock.cc:648] HybridClock initialized: now 1790194703817708 us; error 98 us; skew 500 ppm
I20260923 20:18:23.822671 18025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:23 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194703, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:23.829329 18025 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:18:23.860898 18025 webserver.cc:460] Webserver started at http://127.5.83.66:42819/ using document root <none> and password file <none>
I20260923 20:18:23.862170 18025 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:23.862419 18025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:23.862928 18025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:18:23 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194703, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:18:23.902009 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "131fad945eed4a6384de6f64f939460b"
format_stamp: "Formatted at 2026-09-23 20:18:23 on dist-test-slave-vl24"
server_key: "193357b9fb5a385a8cd01f29a2f6464d"
server_key_iv: "63d6cefc3c98d8f9354d9871fa6b5fec"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:23.903298 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "131fad945eed4a6384de6f64f939460b"
format_stamp: "Formatted at 2026-09-23 20:18:23 on dist-test-slave-vl24"
server_key: "193357b9fb5a385a8cd01f29a2f6464d"
server_key_iv: "63d6cefc3c98d8f9354d9871fa6b5fec"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:23.917176 18025 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.004s
I20260923 20:18:23.965232 18043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:23.967550 18025 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.005s sys 0.005s
I20260923 20:18:23.967875 18025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "131fad945eed4a6384de6f64f939460b"
format_stamp: "Formatted at 2026-09-23 20:18:23 on dist-test-slave-vl24"
server_key: "193357b9fb5a385a8cd01f29a2f6464d"
server_key_iv: "63d6cefc3c98d8f9354d9871fa6b5fec"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:23.968219 18025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:23.990142 18025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:23.995262 18025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:23.995579 18025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:23.996807 18025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:23.998636 18025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:23.998733 18025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:23.998920 18025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:23.999012 18025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:24.053426 18025 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:37149
I20260923 20:18:24.053474 18161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:37149 every 8 connection(s)
I20260923 20:18:24.055117 18025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:18:24.059700 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 18025
I20260923 20:18:24.059904 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:18:24.060640 5453 external_mini_cluster.cc:1428] Setting key 33197d93d1701270a6fa350388dc6c67
Sep 23 20:18:24 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194703, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:24.088564 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:24.057986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:50763 (local address 127.5.83.126:44663)
0923 20:18:24.066631 (+ 8645us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:24.066637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:24.066660 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:18:24.069306 (+ 2646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:24.069312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:24.069413 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:24.069517 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:24.071649 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.072797 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.075371 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.076001 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.076108 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:24.080878 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:24.080885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:24.080890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:24.080932 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:24.084399 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:24.085436 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:24.085442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:24.085445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:24.085610 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:24.086213 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:24.086218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:24.086222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:24.086577 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:24.086704 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:24.087760 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0923 20:18:24.088061 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8526,"thread_start_us":14543,"threads_started":1}
I20260923 20:18:24.091348 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:24.058837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:44663 (local address 127.5.83.66:50763)
0923 20:18:24.065136 (+ 6299us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:24.065186 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:24.068490 (+ 3304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:24.069971 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:24.069984 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:24.070610 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:24.071439 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:24.071463 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.073024 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.073029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:24.075190 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:24.075206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.076460 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.076467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:24.076798 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:24.077623 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:24.077654 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:24.080566 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:24.084662 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:24.084672 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:24.084692 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:24.085291 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:24.085773 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:24.085778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:24.085782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:24.086082 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:24.086795 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:24.086803 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:24.087274 (+ 471us) client_negotiation.cc:769] Sending connection context
0923 20:18:24.089132 (+ 1858us) client_negotiation.cc:240] Negotiation successful
0923 20:18:24.089800 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5937,"thread_start_us":13563,"threads_started":1}
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
I20260923 20:18:24.093349 18162 heartbeater.cc:347] Connected to a master server at 127.5.83.126:44663
I20260923 20:18:24.093722 18162 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:24.094463 18162 heartbeater.cc:511] Master 127.5.83.126:44663 requested a full tablet report, sending...
I20260923 20:18:24.095901 17806 ts_manager.cc:194] Registered new tserver with Master: 131fad945eed4a6384de6f64f939460b (127.5.83.66:37149)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
I20260923 20:18:24.097167 17806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:50763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:18:24.132457 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.84:49651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:44663
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:18:24.249323 18019 heartbeater.cc:503] Master 127.5.83.126:44663 was elected leader, sending a full tablet report...
W20260923 20:18:24.538695 18168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:24.538976 18168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:24.539126 18168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:24.546504 18168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:24.546607 18168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:24.546767 18168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:18:24.556740 18168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.83.84:49651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:44663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:24.558050 18168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:24.559334 18168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:24.572445 18174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:24.572474 18173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:24.572587 18176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:24.573271 18168 server_base.cc:1034] running on GCE node
I20260923 20:18:24.573968 18168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:24.575105 18168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:24.576326 18168 hybrid_clock.cc:648] HybridClock initialized: now 1790194704576294 us; error 46 us; skew 500 ppm
I20260923 20:18:24.576716 18168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:24 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194704, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:24.580147 18168 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:18:24.581804 18168 webserver.cc:460] Webserver started at http://127.5.83.67:46289/ using document root <none> and password file <none>
I20260923 20:18:24.582453 18168 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:24.582551 18168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:24.582799 18168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 20:18:24 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194704, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:18:24.601503 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "edf093d5ab354de7b497d178c2494950"
format_stamp: "Formatted at 2026-09-23 20:18:24 on dist-test-slave-vl24"
server_key: "73290448de9ea5cfa06d213a9f7d724f"
server_key_iv: "9963ec277794e9f9ccf17e1d33f04763"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:24.602641 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "edf093d5ab354de7b497d178c2494950"
format_stamp: "Formatted at 2026-09-23 20:18:24 on dist-test-slave-vl24"
server_key: "73290448de9ea5cfa06d213a9f7d724f"
server_key_iv: "9963ec277794e9f9ccf17e1d33f04763"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:24.609637 18168 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260923 20:18:24.629109 18183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:24.630827 18168 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 20:18:24.631049 18168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "edf093d5ab354de7b497d178c2494950"
format_stamp: "Formatted at 2026-09-23 20:18:24 on dist-test-slave-vl24"
server_key: "73290448de9ea5cfa06d213a9f7d724f"
server_key_iv: "9963ec277794e9f9ccf17e1d33f04763"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:24.631269 18168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:24.653046 18168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:24.658890 18168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:24.659262 18168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:24.660566 18168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:24.662693 18168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:24.662801 18168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:24.662938 18168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:24.663020 18168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:24.698784 18168 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:44079
I20260923 20:18:24.698832 18295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:44079 every 8 connection(s)
I20260923 20:18:24.700038 18168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:18:24.707716 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 18168
I20260923 20:18:24.707934 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:18:24.708478 5453 external_mini_cluster.cc:1428] Setting key 59032e62f4b48fe58a470b10b5575865
Sep 23 20:18:24 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194704, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:24.719853 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:24.702662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:42217 (local address 127.5.83.126:44663)
0923 20:18:24.703188 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:24.703192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:24.703987 (+ 795us) server_negotiation.cc:407] Connection header received
0923 20:18:24.705872 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:24.705877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:24.705982 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:24.706107 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:24.707534 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.708462 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.710358 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.710991 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.711118 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:24.714487 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:24.714491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:24.714495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:24.714525 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:24.717234 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:24.717888 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:24.717891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:24.717893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:24.718012 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:24.718434 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:24.718437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:24.718439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:24.718707 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:24.718814 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:24.719414 (+ 600us) server_negotiation.cc:299] Negotiation successful
0923 20:18:24.719580 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":292,"threads_started":1}
I20260923 20:18:24.720953 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:24.703046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:44663 (local address 127.5.83.67:42217)
0923 20:18:24.703818 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:24.703860 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:24.705577 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:24.706361 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:24.706369 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:24.706783 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:24.707359 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:24.707377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.708690 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.708694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:24.710207 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:24.710219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.711221 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.711226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:24.711534 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:24.712151 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:24.712173 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:24.714289 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:24.717353 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:24.717359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:24.717374 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:24.717771 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:24.718104 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:24.718107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:24.718109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:24.718342 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:24.718813 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:24.718818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:24.719121 (+ 303us) client_negotiation.cc:769] Sending connection context
0923 20:18:24.719339 (+ 218us) client_negotiation.cc:240] Negotiation successful
0923 20:18:24.719696 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":311,"threads_started":1}
I20260923 20:18:24.722371 18296 heartbeater.cc:347] Connected to a master server at 127.5.83.126:44663
I20260923 20:18:24.722704 18296 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:24.723505 18296 heartbeater.cc:511] Master 127.5.83.126:44663 requested a full tablet report, sending...
I20260923 20:18:24.724925 17806 ts_manager.cc:194] Registered new tserver with Master: edf093d5ab354de7b497d178c2494950 (127.5.83.67:44079)
I20260923 20:18:24.725684 17806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:42217
I20260923 20:18:24.736346 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:18:24.738137 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 17886
Sep 23 20:18:24 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:18:24.754499 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:24.739169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.5.83.65:44397)
0923 20:18:24.739785 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:24.739792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:24.739822 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:18:24.739877 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:24.739883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:24.740081 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:24.740204 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:24.741734 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.742756 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.744192 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.744874 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.744979 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:24.749204 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:24.749224 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:24.749238 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:24.749278 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:24.751929 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:24.752659 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:24.752666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:24.752669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:24.752832 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:24.753166 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:24.753170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:24.753171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:24.753510 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:24.753753 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:24.754020 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 20:18:24.754220 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":302,"threads_started":1}
I20260923 20:18:24.755847 18014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:24.839413 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 17886
I20260923 20:18:24.855580 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 18025
Sep 23 20:18:24 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:18:24.872223 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:24.856472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.5.83.66:37149)
0923 20:18:24.857068 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:24.857075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:24.857102 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:18:24.857168 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:24.857175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:24.857379 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:24.857520 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:24.858826 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.859815 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.861358 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.861908 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.862005 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:24.866174 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:24.866185 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:24.866198 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:24.866250 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:24.869289 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:24.870061 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:24.870068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:24.870070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:24.870220 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:24.870658 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:24.870662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:24.870664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:24.871040 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:24.871266 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:24.871548 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 20:18:24.871767 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":345,"threads_started":1}
I20260923 20:18:24.873569 18157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:24.955621 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 18025
I20260923 20:18:24.971849 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 18168
Sep 23 20:18:24 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:18:24.988690 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:24.972997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.5.83.67:44079)
0923 20:18:24.973686 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:24.973693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:24.973722 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:18:24.973781 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:24.973788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:24.974015 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:24.974142 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:24.975349 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.976607 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.977988 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:24.978592 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:24.978701 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:24.983043 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:24.983054 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:24.983069 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:24.983106 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:24.986058 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:24.986746 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:24.986753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:24.986755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:24.986903 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:24.987225 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:24.987228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:24.987230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:24.987557 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:24.987809 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:24.988046 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 20:18:24.988235 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":396,"threads_started":1}
I20260923 20:18:24.990085 18291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:25.074435 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 18168
I20260923 20:18:25.091054 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 17774
I20260923 20:18:25.092408 17836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:25.205067 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 17774
I20260923 20:18:25.227170 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:44663
--webserver_interface=127.5.83.126
--webserver_port=46511
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:44663
--ranger_config_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.84:49651/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.83.84:49651/kms with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:18:25.559113 18308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:25.559404 18308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:25.559628 18308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:25.567055 18308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:18:25.567169 18308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:25.567268 18308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:25.567343 18308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:18:25.567420 18308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:18:25.578138 18308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.83.84:49651/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:44663
--ranger_config_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=46511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:25.579489 18308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:25.580950 18308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:25.593044 18316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:25.593133 18314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:25.593077 18313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:25.594038 18308 server_base.cc:1034] running on GCE node
I20260923 20:18:25.594831 18308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:25.596312 18308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:25.597546 18308 hybrid_clock.cc:648] HybridClock initialized: now 1790194705597478 us; error 84 us; skew 500 ppm
I20260923 20:18:25.597922 18308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:25 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194705, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:25.601032 18308 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:18:25.602496 18308 webserver.cc:460] Webserver started at http://127.5.83.126:46511/ using document root <none> and password file <none>
I20260923 20:18:25.603165 18308 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:25.603312 18308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:18:25 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194705, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:18:25.652722 18308 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:18:25.655629 18324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:25.657022 18308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:18:25.657212 18308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "aae82e024955424ba5a05b688878d91c"
format_stamp: "Formatted at 2026-09-23 20:18:21 on dist-test-slave-vl24"
server_key: "5be2fd525b66b3ad6af34b7dbb9b1bd0"
server_key_iv: "82669ea49550ec4dfa6476b81cde4ff9"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:25.658195 18308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:25.685468 18308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:25.689610 18308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:25.689925 18308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:25.708797 18308 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:44663
I20260923 20:18:25.708809 18375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:44663 every 8 connection(s)
I20260923 20:18:25.710304 18308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:18:25.711798 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 18308
I20260923 20:18:25.712957 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.84:49651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:44397
--local_ip_for_outbound_sockets=127.5.83.65
--tserver_master_addrs=127.5.83.126:44663
--webserver_port=34205
--webserver_interface=127.5.83.65
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.83.84:49651/kms with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:18:25.720330 18376 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:18:25.724135 18376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c: Bootstrap starting.
I20260923 20:18:25.743144 18376 log.cc:824] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:25.752269 18376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:18:25.752874 18376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c: Bootstrap complete.
I20260923 20:18:25.762223 18376 raft_consensus.cc:348] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae82e024955424ba5a05b688878d91c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 44663 } }
I20260923 20:18:25.763257 18376 raft_consensus.cc:729] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae82e024955424ba5a05b688878d91c, State: Initialized, Role: FOLLOWER
I20260923 20:18:25.763859 18376 consensus_queue.cc:260] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae82e024955424ba5a05b688878d91c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 44663 } }
I20260923 20:18:25.764106 18376 raft_consensus.cc:388] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:18:25.764247 18376 raft_consensus.cc:482] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:18:25.764423 18376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:18:25.768237 18376 raft_consensus.cc:504] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae82e024955424ba5a05b688878d91c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 44663 } }
I20260923 20:18:25.768821 18376 leader_election.cc:302] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aae82e024955424ba5a05b688878d91c; no voters:
I20260923 20:18:25.769462 18376 leader_election.cc:288] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:18:25.769598 18380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:18:25.770798 18380 raft_consensus.cc:686] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [term 2 LEADER]: Becoming Leader. State: Replica: aae82e024955424ba5a05b688878d91c, State: Running, Role: LEADER
I20260923 20:18:25.771430 18380 consensus_queue.cc:237] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae82e024955424ba5a05b688878d91c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 44663 } }
I20260923 20:18:25.771803 18376 sys_catalog.cc:564] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:18:25.774003 18382 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aae82e024955424ba5a05b688878d91c. Latest consensus state: current_term: 2 leader_uuid: "aae82e024955424ba5a05b688878d91c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae82e024955424ba5a05b688878d91c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 44663 } } }
I20260923 20:18:25.774461 18382 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:25.775085 18381 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aae82e024955424ba5a05b688878d91c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae82e024955424ba5a05b688878d91c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 44663 } } }
I20260923 20:18:25.775415 18381 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:25.781739 18386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:18:25.788255 18386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:18:25.789821 18386 catalog_manager.cc:1259] Loaded cluster ID: 3217dda913af4dd1bf46452d5121f923
I20260923 20:18:25.789980 18386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:18:25.794176 18386 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:18:25.795678 18386 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c: Loaded TSK: 0
I20260923 20:18:25.797418 18386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:18:25.800279 18376 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260923 20:18:26.171530 18378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:26.172068 18378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:26.172268 18378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:26.183841 18378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:26.184103 18378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:26.184351 18378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:18:26.197561 18378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.83.84:49651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:44397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=34205
--tserver_master_addrs=127.5.83.126:44663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:26.199617 18378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:26.201514 18378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:26.232357 18415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:26.239174 18418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:26.240123 18378 server_base.cc:1034] running on GCE node
W20260923 20:18:26.240845 18414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:26.241812 18378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:26.243086 18378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:26.244388 18378 hybrid_clock.cc:648] HybridClock initialized: now 1790194706244278 us; error 101 us; skew 500 ppm
I20260923 20:18:26.244979 18378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:26 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:26.248874 18378 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:18:26.251976 18378 webserver.cc:460] Webserver started at http://127.5.83.65:34205/ using document root <none> and password file <none>
I20260923 20:18:26.253196 18378 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:26.253427 18378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:18:26 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:18:26.306651 18378 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.005s sys 0.012s
I20260923 20:18:26.310868 18425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:26.312997 18378 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260923 20:18:26.313258 18378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "4bed110f85004da19bc137ba4c417751"
format_stamp: "Formatted at 2026-09-23 20:18:23 on dist-test-slave-vl24"
server_key: "eb665aa75e777f01f3da5f7759a5d629"
server_key_iv: "43f53f239166df53cb340716dc8bb8b0"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:26.314882 18378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:26.339941 18378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:26.345399 18378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:26.345726 18378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:26.346987 18378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:26.348832 18378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:26.348949 18378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:26.349062 18378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:26.349138 18378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:26.416158 18378 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:44397
I20260923 20:18:26.416244 18538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:44397 every 8 connection(s)
I20260923 20:18:26.418109 18378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:18:26.419636 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 18378
I20260923 20:18:26.420499 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.84:49651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:37149
--local_ip_for_outbound_sockets=127.5.83.66
--tserver_master_addrs=127.5.83.126:44663
--webserver_port=42819
--webserver_interface=127.5.83.66
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.83.84:49651/kms with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:18:26 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:26.510454 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:26.444413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:39129 (local address 127.5.83.126:44663)
0923 20:18:26.467933 (+ 23520us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:26.467959 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:26.472293 (+ 4334us) server_negotiation.cc:407] Connection header received
0923 20:18:26.478809 (+ 6516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:26.478836 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:26.479413 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:26.479832 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:26.486703 (+ 6871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:26.488795 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:26.491396 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:26.492708 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:26.492964 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:26.498424 (+ 5460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:26.498436 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:26.498460 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:26.498514 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:26.503502 (+ 4988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:26.504659 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:26.504669 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:26.504678 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:26.504890 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:26.505451 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:26.505457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:26.505461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:26.506130 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:26.506307 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:26.508729 (+ 2422us) server_negotiation.cc:299] Negotiation successful
0923 20:18:26.509175 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23152,"thread_start_us":23445,"threads_started":1}
I20260923 20:18:26.512100 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:26.443130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:44663 (local address 127.5.83.65:39129)
0923 20:18:26.468906 (+ 25776us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:26.468977 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:26.477912 (+ 8935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:26.480739 (+ 2827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:26.480751 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:26.481514 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:26.482395 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:26.482425 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:26.489085 (+ 6660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:26.489091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:26.491178 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:26.491210 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:26.493497 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:26.493506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:26.493869 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:26.494744 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:26.494783 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:26.497923 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:26.503729 (+ 5806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:26.503741 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:26.503759 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:26.504450 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:26.505039 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:26.505045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:26.505049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:26.505284 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:26.506403 (+ 1119us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:26.506412 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:26.506814 (+ 402us) client_negotiation.cc:769] Sending connection context
0923 20:18:26.507100 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 20:18:26.507698 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25413,"thread_start_us":33291,"threads_started":1}
I20260923 20:18:26.515956 18539 heartbeater.cc:347] Connected to a master server at 127.5.83.126:44663
I20260923 20:18:26.516283 18539 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:26.517182 18539 heartbeater.cc:511] Master 127.5.83.126:44663 requested a full tablet report, sending...
W20260923 20:18:26.520515 18539 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:44663 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:18:26.520810 18539 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:26.521106 18539 heartbeater.cc:511] Master 127.5.83.126:44663 requested a full tablet report, sending...
I20260923 20:18:26.522771 18539 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:26.523101 18539 heartbeater.cc:511] Master 127.5.83.126:44663 requested a full tablet report, sending...
W20260923 20:18:26.524286 18539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:18:26.919749 18541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:26.920100 18541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:26.920277 18541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:26.928304 18541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:26.928462 18541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:26.928689 18541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:18:26.942597 18541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.83.84:49651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:37149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=42819
--tserver_master_addrs=127.5.83.126:44663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:26.944166 18541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:26.945797 18541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:26.963392 18558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:26.964604 18557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:26.964771 18560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:26.966395 18541 server_base.cc:1034] running on GCE node
I20260923 20:18:26.967175 18541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:26.968406 18541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:26.969738 18541 hybrid_clock.cc:648] HybridClock initialized: now 1790194706969679 us; error 50 us; skew 500 ppm
I20260923 20:18:26.970093 18541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:26 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:26.976696 18541 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:18:26.978677 18541 webserver.cc:460] Webserver started at http://127.5.83.66:42819/ using document root <none> and password file <none>
I20260923 20:18:26.979527 18541 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:26.979724 18541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:18:26 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:18:27.003966 18541 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:18:27.010509 18568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:27.012789 18541 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.002s
I20260923 20:18:27.013108 18541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "131fad945eed4a6384de6f64f939460b"
format_stamp: "Formatted at 2026-09-23 20:18:23 on dist-test-slave-vl24"
server_key: "193357b9fb5a385a8cd01f29a2f6464d"
server_key_iv: "63d6cefc3c98d8f9354d9871fa6b5fec"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:27.014725 18541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:27.067004 18541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:27.072306 18541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:27.072731 18541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:27.073998 18541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:27.075794 18541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:27.075927 18541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:27.076077 18541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:27.076177 18541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:27.133767 18541 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:37149
I20260923 20:18:27.133944 18682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:37149 every 8 connection(s)
I20260923 20:18:27.135444 18541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
Sep 23 20:18:27 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790194707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:27.144549 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 18541
I20260923 20:18:27.145601 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.83.84:49651/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:44079
--local_ip_for_outbound_sockets=127.5.83.67
--tserver_master_addrs=127.5.83.126:44663
--webserver_port=46289
--webserver_interface=127.5.83.67
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.83.84:49651/kms with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:18:27 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:27.189702 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:27.139413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:50849 (local address 127.5.83.126:44663)
0923 20:18:27.152311 (+ 12898us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:27.152320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:27.157178 (+ 4858us) server_negotiation.cc:407] Connection header received
0923 20:18:27.157269 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:27.157273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:27.157359 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:27.157503 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:27.159704 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:27.172101 (+ 12397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:27.175608 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:27.176495 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:27.176770 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:27.182081 (+ 5311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:27.182092 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:27.182100 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:27.182154 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:27.185906 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:27.186940 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:27.186946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:27.186950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:27.187097 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:27.187589 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:27.187594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:27.187598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:27.187951 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:27.188085 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:27.189084 (+ 999us) server_negotiation.cc:299] Negotiation successful
0923 20:18:27.189324 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12713,"thread_start_us":13132,"threads_started":1}
I20260923 20:18:27.192485 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:27.140242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:44663 (local address 127.5.83.66:50849)
0923 20:18:27.153760 (+ 13518us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:27.153834 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:27.156729 (+ 2895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:27.157830 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:27.157845 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:27.158555 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:27.159451 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:27.159481 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:27.172454 (+ 12973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:27.172464 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:27.175316 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:27.175347 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:27.177214 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:27.177225 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:27.177608 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:27.178533 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:27.178582 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:27.181723 (+ 3141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:27.186111 (+ 4388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:27.186122 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:27.186143 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:27.186774 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:27.187217 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:27.187222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:27.187226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:27.187462 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:27.188167 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:27.188177 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:27.188658 (+ 481us) client_negotiation.cc:769] Sending connection context
0923 20:18:27.190380 (+ 1722us) client_negotiation.cc:240] Negotiation successful
0923 20:18:27.191056 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13213,"thread_start_us":12191,"threads_started":1}
I20260923 20:18:27.194483 18683 heartbeater.cc:347] Connected to a master server at 127.5.83.126:44663
I20260923 20:18:27.194796 18683 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:27.195647 18683 heartbeater.cc:511] Master 127.5.83.126:44663 requested a full tablet report, sending...
I20260923 20:18:27.197856 18341 ts_manager.cc:194] Registered new tserver with Master: 131fad945eed4a6384de6f64f939460b (127.5.83.66:37149)
I20260923 20:18:27.199920 18341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:50849
I20260923 20:18:27.524464 18539 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:27.524995 18539 heartbeater.cc:511] Master 127.5.83.126:44663 requested a full tablet report, sending...
I20260923 20:18:27.526432 18341 ts_manager.cc:194] Registered new tserver with Master: 4bed110f85004da19bc137ba4c417751 (127.5.83.65:44397)
I20260923 20:18:27.527693 18341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:39129
W20260923 20:18:27.595410 18687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:27.595729 18687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:27.595918 18687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:27.605432 18687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:27.605580 18687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:27.605772 18687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:18:27.620153 18687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.83.84:49651/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:44079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=46289
--tserver_master_addrs=127.5.83.126:44663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:27.622306 18687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:27.624127 18687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 23 20:18:27 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
W20260923 20:18:27.642379 18693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:27.642352 18694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:27.654054 18696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:27.662076 18687 server_base.cc:1034] running on GCE node
I20260923 20:18:27.663075 18687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:27.664808 18687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:27.667006 18687 hybrid_clock.cc:648] HybridClock initialized: now 1790194707666922 us; error 61 us; skew 500 ppm
I20260923 20:18:27.667647 18687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:27 dist-test-slave-vl24 krb5kdc[17330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:27.672931 18687 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:18:27.718798 18687 webserver.cc:460] Webserver started at http://127.5.83.67:46289/ using document root <none> and password file <none>
I20260923 20:18:27.719934 18687 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:27.720180 18687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 20:18:27 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:18:27.770519 18687 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.007s sys 0.012s
I20260923 20:18:27.776494 18704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:27.778489 18687 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260923 20:18:27.778687 18687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "edf093d5ab354de7b497d178c2494950"
format_stamp: "Formatted at 2026-09-23 20:18:24 on dist-test-slave-vl24"
server_key: "73290448de9ea5cfa06d213a9f7d724f"
server_key_iv: "9963ec277794e9f9ccf17e1d33f04763"
server_key_version: "kuduclusterkey@0"
I20260923 20:18:27.780083 18687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:27.845366 18687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:27.850447 18687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:27.850723 18687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:27.851898 18687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:27.853574 18687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:27.853682 18687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:27.853796 18687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:27.853883 18687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 20:18:27 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for HTTP/127.5.83.84@KRBTEST.COM
I20260923 20:18:28.106653 18817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:44079 every 8 connection(s)
I20260923 20:18:28.106995 18687 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:44079
I20260923 20:18:28.108969 18687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:18:28.114840 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 18687
Sep 23 20:18:28 dist-test-slave-vl24 krb5kdc[17330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:28.179409 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.115883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.5.83.126:44663)
0923 20:18:28.159894 (+ 44011us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:28.159902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:28.159929 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:18:28.160016 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:28.160023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:28.160123 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:28.160273 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:28.163039 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.164115 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.166214 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.166822 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.166959 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.170377 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:28.170384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:28.170389 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:28.170429 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:28.174294 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:28.175400 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:28.175405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:28.175409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:28.175536 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:28.176574 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:28.176579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:28.176583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:28.176925 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:28.177193 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:28.178760 (+ 1567us) server_negotiation.cc:299] Negotiation successful
0923 20:18:28.179012 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43829,"thread_start_us":48393,"threads_started":1}
I20260923 20:18:28.188937 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.132947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:45731 (local address 127.5.83.126:44663)
0923 20:18:28.162557 (+ 29610us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:28.162563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:28.162582 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:18:28.162640 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:28.162645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:28.162720 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:28.162819 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:28.167498 (+ 4679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.168654 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.174465 (+ 5811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.175127 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.175213 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.180974 (+ 5761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:28.180980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:28.180985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:28.181019 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:28.185136 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:28.186202 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:28.186207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:28.186210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:28.186353 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:28.186897 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:28.186902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:28.186906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:28.187234 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:28.187366 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:28.188288 (+ 922us) server_negotiation.cc:299] Negotiation successful
0923 20:18:28.188507 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29461,"thread_start_us":28336,"threads_started":1}
I20260923 20:18:28.190331 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.132182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:44663 (local address 127.5.83.67:45731)
0923 20:18:28.156330 (+ 24148us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.156406 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:28.158905 (+ 2499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:28.164726 (+ 5821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:28.164739 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:28.165370 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:28.167267 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:28.167299 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.168847 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.168865 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:28.172721 (+ 3856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:28.172750 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.175820 (+ 3070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.175826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:28.176145 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.177359 (+ 1214us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:28.177390 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:28.180677 (+ 3287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:28.185292 (+ 4615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:28.185304 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:28.185324 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:28.186020 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:28.186496 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:28.186501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:28.186505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:28.186752 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:28.187389 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:28.187397 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:28.187834 (+ 437us) client_negotiation.cc:769] Sending connection context
0923 20:18:28.188136 (+ 302us) client_negotiation.cc:240] Negotiation successful
0923 20:18:28.188749 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23761,"thread_start_us":28127,"threads_started":1}
I20260923 20:18:28.191962 18818 heartbeater.cc:347] Connected to a master server at 127.5.83.126:44663
I20260923 20:18:28.192301 18818 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:28.193173 18818 heartbeater.cc:511] Master 127.5.83.126:44663 requested a full tablet report, sending...
I20260923 20:18:28.195955 18341 ts_manager.cc:194] Registered new tserver with Master: edf093d5ab354de7b497d178c2494950 (127.5.83.67:44079)
I20260923 20:18:28.197221 18341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:45731
I20260923 20:18:28.204665 18683 heartbeater.cc:503] Master 127.5.83.126:44663 was elected leader, sending a full tablet report...
I20260923 20:18:28.208195 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:18:28.236506 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.218832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.5.83.126:44663)
0923 20:18:28.219176 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:28.219183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:28.219542 (+ 359us) server_negotiation.cc:407] Connection header received
0923 20:18:28.219721 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:28.219727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:28.219822 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:28.219940 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:28.222155 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.223356 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.225615 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.226305 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.226435 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.229785 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:28.229792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:28.229797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:28.229840 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:28.233402 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:28.234290 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:28.234295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:28.234299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:28.234454 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:28.234937 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:28.234943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:28.234946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:28.235285 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:28.235523 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:28.235933 (+ 410us) server_negotiation.cc:299] Negotiation successful
0923 20:18:28.236157 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260923 20:18:28.242161 18341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:18:28.249329 18341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:18:28.287426 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.263203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:44079 (local address 127.0.0.1:54126)
0923 20:18:28.277488 (+ 14285us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.277526 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:28.277780 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:28.278871 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:28.278879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:28.278910 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:28.279241 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:28.279249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.281944 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.281949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:28.284502 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:28.284522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.286490 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.286496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:28.286703 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.286749 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 20:18:28.286835 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 20:18:28.286963 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14047,"mutex_wait_us":62,"thread_start_us":14792,"threads_started":1}
I20260923 20:18:28.288163 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.263763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.5.83.65:44397)
0923 20:18:28.279210 (+ 15447us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:28.279218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:28.280225 (+ 1007us) server_negotiation.cc:407] Connection header received
0923 20:18:28.280350 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:28.280383 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:28.280701 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:28.280916 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:28.281681 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.283300 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.285614 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.287010 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.287146 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.287399 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:28.287667 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 20:18:28.287841 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15279,"thread_start_us":15527,"threads_started":1}
I20260923 20:18:28.288436 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.263586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.5.83.67:44079)
0923 20:18:28.276438 (+ 12852us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:28.276452 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:28.277668 (+ 1216us) server_negotiation.cc:407] Connection header received
0923 20:18:28.278052 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:28.278060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:28.278342 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:28.278528 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:28.279527 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.281769 (+ 2242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.284676 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.286254 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.287508 (+ 1254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.287744 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:28.287805 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 20:18:28.287959 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12589,"thread_start_us":12905,"threads_started":1}
I20260923 20:18:28.288945 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.263721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44397 (local address 127.0.0.1:60516)
0923 20:18:28.279938 (+ 16217us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.279964 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:28.280083 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:28.281001 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:28.281006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:28.281031 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:28.281470 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:28.281485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.283439 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.283444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:28.285415 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:28.285428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.287232 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.287237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:28.287433 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.287475 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 20:18:28.288408 (+ 933us) client_negotiation.cc:240] Negotiation successful
0923 20:18:28.288575 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16092,"mutex_wait_us":1,"thread_start_us":15959,"threads_started":1}
I20260923 20:18:28.289896 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.280094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:37149 (local address 127.0.0.1:43156)
0923 20:18:28.281756 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.281774 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:28.281890 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:28.282764 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:28.282769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:28.282792 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:28.283153 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:28.283163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.284964 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.284970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:28.287232 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:28.287245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.289120 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.289126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:28.289302 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.289332 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:18:28.289459 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 20:18:28.289576 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1422,"thread_start_us":1258,"threads_started":1}
I20260923 20:18:28.290818 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.280312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.5.83.66:37149)
0923 20:18:28.280907 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:28.280915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:28.282032 (+ 1117us) server_negotiation.cc:407] Connection header received
0923 20:18:28.282136 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:28.282143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:28.282349 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:28.282525 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:28.283369 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.284836 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.287467 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.288890 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.289042 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.289260 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:28.289500 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 20:18:28.290491 (+ 991us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":282,"threads_started":1}
I20260923 20:18:28.292459 18752 tablet_service.cc:1467] Processing CreateTablet for tablet 174829fc6aee4927b863f7e3e2131358 (DEFAULT_TABLE table=test-table [id=2c5117ce5f7a42d6ae3bc64ae1306290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:18:28.292685 18473 tablet_service.cc:1467] Processing CreateTablet for tablet 174829fc6aee4927b863f7e3e2131358 (DEFAULT_TABLE table=test-table [id=2c5117ce5f7a42d6ae3bc64ae1306290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:18:28.294034 18752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174829fc6aee4927b863f7e3e2131358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:28.294086 18473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174829fc6aee4927b863f7e3e2131358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:28.294250 18618 tablet_service.cc:1467] Processing CreateTablet for tablet 174829fc6aee4927b863f7e3e2131358 (DEFAULT_TABLE table=test-table [id=2c5117ce5f7a42d6ae3bc64ae1306290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:18:28.295569 18618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174829fc6aee4927b863f7e3e2131358. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:28.307108 18846 tablet_bootstrap.cc:492] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950: Bootstrap starting.
I20260923 20:18:28.311930 18846 tablet_bootstrap.cc:654] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:28.313482 18846 log.cc:824] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:28.313758 18847 tablet_bootstrap.cc:492] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751: Bootstrap starting.
I20260923 20:18:28.316313 18848 tablet_bootstrap.cc:492] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b: Bootstrap starting.
I20260923 20:18:28.317188 18846 tablet_bootstrap.cc:492] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950: No bootstrap required, opened a new log
I20260923 20:18:28.317462 18846 ts_tablet_manager.cc:1397] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 20:18:28.317613 18847 tablet_bootstrap.cc:654] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:28.318902 18847 log.cc:824] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:28.320117 18848 tablet_bootstrap.cc:654] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:28.321408 18848 log.cc:824] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:28.322316 18847 tablet_bootstrap.cc:492] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751: No bootstrap required, opened a new log
I20260923 20:18:28.322626 18847 ts_tablet_manager.cc:1397] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 20:18:28.324604 18848 tablet_bootstrap.cc:492] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b: No bootstrap required, opened a new log
I20260923 20:18:28.324863 18848 ts_tablet_manager.cc:1397] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 20:18:28.326586 18846 raft_consensus.cc:348] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf093d5ab354de7b497d178c2494950" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44079 } } peers { permanent_uuid: "4bed110f85004da19bc137ba4c417751" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44397 } } peers { permanent_uuid: "131fad945eed4a6384de6f64f939460b" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37149 } }
I20260923 20:18:28.326965 18846 raft_consensus.cc:374] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:28.327067 18846 raft_consensus.cc:729] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf093d5ab354de7b497d178c2494950, State: Initialized, Role: FOLLOWER
I20260923 20:18:28.327684 18846 consensus_queue.cc:260] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf093d5ab354de7b497d178c2494950" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44079 } } peers { permanent_uuid: "4bed110f85004da19bc137ba4c417751" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44397 } } peers { permanent_uuid: "131fad945eed4a6384de6f64f939460b" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37149 } }
I20260923 20:18:28.329854 18818 heartbeater.cc:503] Master 127.5.83.126:44663 was elected leader, sending a full tablet report...
I20260923 20:18:28.331144 18846 ts_tablet_manager.cc:1428] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260923 20:18:28.333295 18847 raft_consensus.cc:348] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf093d5ab354de7b497d178c2494950" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44079 } } peers { permanent_uuid: "4bed110f85004da19bc137ba4c417751" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44397 } } peers { permanent_uuid: "131fad945eed4a6384de6f64f939460b" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37149 } }
I20260923 20:18:28.333738 18847 raft_consensus.cc:374] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:28.333892 18847 raft_consensus.cc:729] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bed110f85004da19bc137ba4c417751, State: Initialized, Role: FOLLOWER
I20260923 20:18:28.334563 18847 consensus_queue.cc:260] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf093d5ab354de7b497d178c2494950" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44079 } } peers { permanent_uuid: "4bed110f85004da19bc137ba4c417751" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44397 } } peers { permanent_uuid: "131fad945eed4a6384de6f64f939460b" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37149 } }
I20260923 20:18:28.335304 18848 raft_consensus.cc:348] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf093d5ab354de7b497d178c2494950" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44079 } } peers { permanent_uuid: "4bed110f85004da19bc137ba4c417751" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44397 } } peers { permanent_uuid: "131fad945eed4a6384de6f64f939460b" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37149 } }
I20260923 20:18:28.335764 18848 raft_consensus.cc:374] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:28.335819 18539 heartbeater.cc:503] Master 127.5.83.126:44663 was elected leader, sending a full tablet report...
I20260923 20:18:28.335866 18848 raft_consensus.cc:729] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131fad945eed4a6384de6f64f939460b, State: Initialized, Role: FOLLOWER
I20260923 20:18:28.336505 18847 ts_tablet_manager.cc:1428] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260923 20:18:28.336484 18848 consensus_queue.cc:260] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf093d5ab354de7b497d178c2494950" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44079 } } peers { permanent_uuid: "4bed110f85004da19bc137ba4c417751" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44397 } } peers { permanent_uuid: "131fad945eed4a6384de6f64f939460b" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37149 } }
I20260923 20:18:28.338112 18848 ts_tablet_manager.cc:1428] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260923 20:18:28.397902 18684 tablet.cc:2369] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:18:28.412518 18821 tablet.cc:2369] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:18:28.431983 18854 raft_consensus.cc:482] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:18:28.432322 18854 raft_consensus.cc:504] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf093d5ab354de7b497d178c2494950" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44079 } } peers { permanent_uuid: "4bed110f85004da19bc137ba4c417751" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44397 } } peers { permanent_uuid: "131fad945eed4a6384de6f64f939460b" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37149 } }
I20260923 20:18:28.434425 18854 leader_election.cc:288] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edf093d5ab354de7b497d178c2494950 (127.5.83.67:44079), 4bed110f85004da19bc137ba4c417751 (127.5.83.65:44397)
I20260923 20:18:28.444183 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.434890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:56853 (local address 127.5.83.67:44079)
0923 20:18:28.435110 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:28.435117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:28.436081 (+ 964us) server_negotiation.cc:407] Connection header received
0923 20:18:28.436295 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:28.436301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:28.436384 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:28.436486 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:28.437443 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.438534 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.441675 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.442946 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.443085 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.443196 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:28.443790 (+ 594us) server_negotiation.cc:299] Negotiation successful
0923 20:18:28.443921 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:18:28.444185 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.434974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:57605 (local address 127.5.83.65:44397)
0923 20:18:28.435319 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:28.435325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:28.436076 (+ 751us) server_negotiation.cc:407] Connection header received
0923 20:18:28.436295 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:28.436301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:28.436391 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:28.436486 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:28.437470 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.438536 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.441675 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.442946 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.443085 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.443196 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:28.443790 (+ 594us) server_negotiation.cc:299] Negotiation successful
0923 20:18:28.443946 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260923 20:18:28.444511 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.434705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:44079 (local address 127.5.83.66:56853)
0923 20:18:28.435943 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.435965 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:28.436125 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:28.436551 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:28.436557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:28.436606 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:28.437250 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:28.437266 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.438681 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.438685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:28.441465 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:28.441481 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.443261 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.443266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:28.443563 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.443651 (+ 88us) client_negotiation.cc:769] Sending connection context
0923 20:18:28.443748 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:18:28.443986 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":408,"spinlock_wait_cycles":15872,"thread_start_us":828,"threads_started":1}
I20260923 20:18:28.444489 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.434792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44397 (local address 127.5.83.66:57605)
0923 20:18:28.435943 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.435965 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:28.436163 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:28.436551 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:28.436557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:28.436608 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:28.437304 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:28.437315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.438679 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.438685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:28.441465 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:28.441481 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.443225 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.443234 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:28.443563 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.443644 (+ 81us) client_negotiation.cc:769] Sending connection context
0923 20:18:28.443748 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:18:28.443986 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":310,"spinlock_wait_cycles":45184,"thread_start_us":693,"threads_started":1}
I20260923 20:18:28.445581 18772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174829fc6aee4927b863f7e3e2131358" candidate_uuid: "131fad945eed4a6384de6f64f939460b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf093d5ab354de7b497d178c2494950" is_pre_election: true
I20260923 20:18:28.445640 18493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174829fc6aee4927b863f7e3e2131358" candidate_uuid: "131fad945eed4a6384de6f64f939460b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bed110f85004da19bc137ba4c417751" is_pre_election: true
I20260923 20:18:28.446125 18772 raft_consensus.cc:2454] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 131fad945eed4a6384de6f64f939460b in term 0.
I20260923 20:18:28.446125 18493 raft_consensus.cc:2454] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 131fad945eed4a6384de6f64f939460b in term 0.
I20260923 20:18:28.446801 18574 leader_election.cc:302] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 131fad945eed4a6384de6f64f939460b, 4bed110f85004da19bc137ba4c417751, edf093d5ab354de7b497d178c2494950; no voters:
I20260923 20:18:28.447295 18854 raft_consensus.cc:2781] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:18:28.447471 18854 raft_consensus.cc:482] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:18:28.447600 18854 raft_consensus.cc:3037] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:28.449889 18854 raft_consensus.cc:504] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf093d5ab354de7b497d178c2494950" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44079 } } peers { permanent_uuid: "4bed110f85004da19bc137ba4c417751" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44397 } } peers { permanent_uuid: "131fad945eed4a6384de6f64f939460b" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37149 } }
I20260923 20:18:28.450457 18854 leader_election.cc:288] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [CANDIDATE]: Term 1 election: Requested vote from peers edf093d5ab354de7b497d178c2494950 (127.5.83.67:44079), 4bed110f85004da19bc137ba4c417751 (127.5.83.65:44397)
I20260923 20:18:28.450973 18493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174829fc6aee4927b863f7e3e2131358" candidate_uuid: "131fad945eed4a6384de6f64f939460b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bed110f85004da19bc137ba4c417751"
I20260923 20:18:28.450974 18772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174829fc6aee4927b863f7e3e2131358" candidate_uuid: "131fad945eed4a6384de6f64f939460b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edf093d5ab354de7b497d178c2494950"
I20260923 20:18:28.451184 18493 raft_consensus.cc:3037] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:28.451184 18772 raft_consensus.cc:3037] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:28.454407 18493 raft_consensus.cc:2454] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 131fad945eed4a6384de6f64f939460b in term 1.
I20260923 20:18:28.454407 18772 raft_consensus.cc:2454] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 131fad945eed4a6384de6f64f939460b in term 1.
I20260923 20:18:28.454922 18574 leader_election.cc:302] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 131fad945eed4a6384de6f64f939460b, 4bed110f85004da19bc137ba4c417751, edf093d5ab354de7b497d178c2494950; no voters:
I20260923 20:18:28.455228 18854 raft_consensus.cc:2781] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:18:28.455725 18854 raft_consensus.cc:686] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 1 LEADER]: Becoming Leader. State: Replica: 131fad945eed4a6384de6f64f939460b, State: Running, Role: LEADER
I20260923 20:18:28.456308 18854 consensus_queue.cc:237] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf093d5ab354de7b497d178c2494950" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44079 } } peers { permanent_uuid: "4bed110f85004da19bc137ba4c417751" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44397 } } peers { permanent_uuid: "131fad945eed4a6384de6f64f939460b" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37149 } }
I20260923 20:18:28.461038 18340 catalog_manager.cc:5495] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b reported cstate change: term changed from 0 to 1, leader changed from <none> to 131fad945eed4a6384de6f64f939460b (127.5.83.66). New cstate: current_term: 1 leader_uuid: "131fad945eed4a6384de6f64f939460b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edf093d5ab354de7b497d178c2494950" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bed110f85004da19bc137ba4c417751" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "131fad945eed4a6384de6f64f939460b" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37149 } health_report { overall_health: HEALTHY } } }
I20260923 20:18:28.482719 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.477384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.5.83.66:37149)
0923 20:18:28.477570 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:28.477575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:28.477592 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:18:28.477729 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:28.477733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:28.477822 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:28.477948 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:18:28.478278 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.478984 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.480468 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.480922 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.481027 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.481571 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:18:28.482170 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:18:28.482277 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:28.482442 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 20:18:28.482510 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 20:18:28.489241 18540 tablet.cc:2369] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:18:28.491466 18493 raft_consensus.cc:1261] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751 [term 1 FOLLOWER]: Refusing update from remote peer 131fad945eed4a6384de6f64f939460b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:18:28.491477 18772 raft_consensus.cc:1261] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950 [term 1 FOLLOWER]: Refusing update from remote peer 131fad945eed4a6384de6f64f939460b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:18:28.492378 18854 consensus_queue.cc:1035] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [LEADER]: Connected to new peer: Peer: permanent_uuid: "edf093d5ab354de7b497d178c2494950" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 44079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:18:28.492702 18857 consensus_queue.cc:1035] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bed110f85004da19bc137ba4c417751" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:18:28.511412 18862 mvcc.cc:205] Tried to move back new op lower bound from 7332637525970358272 to 7332637525840904192. Current Snapshot: MvccSnapshot[applied={T|T < 7332637525970358272}]
I20260923 20:18:28.527845 18341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:18:28.528169 18341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:18:28.532128 18341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aae82e024955424ba5a05b688878d91c: Sending DeleteTablet for 3 replicas of tablet 174829fc6aee4927b863f7e3e2131358
I20260923 20:18:28.533532 18752 tablet_service.cc:1514] Processing DeleteTablet for tablet 174829fc6aee4927b863f7e3e2131358 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:18:28 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:54126
I20260923 20:18:28.533578 18618 tablet_service.cc:1514] Processing DeleteTablet for tablet 174829fc6aee4927b863f7e3e2131358 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:18:28 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:43156
I20260923 20:18:28.533700 18473 tablet_service.cc:1514] Processing DeleteTablet for tablet 174829fc6aee4927b863f7e3e2131358 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:18:28 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:60516
I20260923 20:18:28.537626 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 18378
I20260923 20:18:28.546169 18870 tablet_replica.cc:330] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950: stopping tablet replica
I20260923 20:18:28.546810 18870 raft_consensus.cc:2229] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:18:28.554226 18870 raft_consensus.cc:2258] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:18:28.556752 18870 ts_tablet_manager.cc:1905] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:18:28.562595 18871 tablet_replica.cc:330] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b: stopping tablet replica
I20260923 20:18:28.563237 18872 tablet_replica.cc:330] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751: stopping tablet replica
I20260923 20:18:28.563772 18872 raft_consensus.cc:2229] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:18:28.564309 18871 raft_consensus.cc:2229] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:18:28.564917 18872 raft_consensus.cc:2258] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:18:28.565119 18870 ts_tablet_manager.cc:1918] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:18:28.565351 18870 log.cc:1196] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/174829fc6aee4927b863f7e3e2131358
I20260923 20:18:28.565688 18871 raft_consensus.cc:2258] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:18:28.565939 18870 ts_tablet_manager.cc:1939] T 174829fc6aee4927b863f7e3e2131358 P edf093d5ab354de7b497d178c2494950: Deleting consensus metadata
I20260923 20:18:28.567859 18327 catalog_manager.cc:4841] TS edf093d5ab354de7b497d178c2494950 (127.5.83.67:44079): tablet 174829fc6aee4927b863f7e3e2131358 (table test-table [id=2c5117ce5f7a42d6ae3bc64ae1306290]) successfully deleted
I20260923 20:18:28.568436 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.538873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.5.83.65:44397)
0923 20:18:28.539475 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:28.539481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:28.539499 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:18:28.539558 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:28.539563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:28.539641 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:28.539741 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:28.548743 (+ 9002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.550020 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.552027 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.552770 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.552904 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.557850 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:28.557877 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:28.557903 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:28.557980 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:28.563636 (+ 5656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:28.564555 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:28.564564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:28.564572 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:28.564742 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:28.565234 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:28.565245 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:28.565250 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:28.565742 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:28.566077 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:28.567847 (+ 1770us) server_negotiation.cc:299] Negotiation successful
0923 20:18:28.568087 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475}
I20260923 20:18:28.568787 18872 ts_tablet_manager.cc:1905] T 174829fc6aee4927b863f7e3e2131358 P 4bed110f85004da19bc137ba4c417751: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:18:28.569130 18534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:28.569480 18871 ts_tablet_manager.cc:1905] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:18:28.578172 18871 ts_tablet_manager.cc:1918] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:18:28.578382 18871 log.cc:1196] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/174829fc6aee4927b863f7e3e2131358
I20260923 20:18:28.578995 18871 ts_tablet_manager.cc:1939] T 174829fc6aee4927b863f7e3e2131358 P 131fad945eed4a6384de6f64f939460b: Deleting consensus metadata
I20260923 20:18:28.580978 18328 catalog_manager.cc:4841] TS 131fad945eed4a6384de6f64f939460b (127.5.83.66:37149): tablet 174829fc6aee4927b863f7e3e2131358 (table test-table [id=2c5117ce5f7a42d6ae3bc64ae1306290]) successfully deleted
I20260923 20:18:28.660573 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 18378
W20260923 20:18:28.683710 18328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.65:44397 (error 108)
I20260923 20:18:28.683923 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 18541
I20260923 20:18:28.685904 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.685211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44397 (local address 127.0.0.1:47438)
0923 20:18:28.685525 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.685734 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
W20260923 20:18:28.686163 18328 catalog_manager.cc:4584] TS 4bed110f85004da19bc137ba4c417751 (127.5.83.65:44397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 174829fc6aee4927b863f7e3e2131358: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44397: connect: Connection refused (error 111)
I20260923 20:18:28.700416 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.685043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.5.83.66:37149)
0923 20:18:28.685223 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:28.685231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:28.685272 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:18:28.685476 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:28.685483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:28.685566 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:28.685679 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:28.687144 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.688390 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.689734 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.690411 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.690535 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.693059 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:28.693076 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:28.693096 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:28.693140 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:28.697394 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:28.698159 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:28.698174 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:28.698178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:28.698356 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:28.698719 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:28.698725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:28.698728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:28.699178 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:28.699528 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:28.699772 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 20:18:28.700031 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:18:28.701264 18678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:28.739883 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.739294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44397 (local address 127.0.0.1:47444)
0923 20:18:28.739575 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.739697 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260923 20:18:28.741107 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.740719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44397 (local address 127.0.0.1:47458)
0923 20:18:28.740875 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.740961 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 20:18:28.801826 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.801284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44397 (local address 127.0.0.1:47460)
0923 20:18:28.801542 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.801649 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260923 20:18:28.802891 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.802514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44397 (local address 127.0.0.1:47464)
0923 20:18:28.802681 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.802763 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 20:18:28.815829 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 18541
I20260923 20:18:28.834789 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 18687
I20260923 20:18:28.848492 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.835694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.5.83.67:44079)
0923 20:18:28.835898 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:28.835903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:28.835949 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:18:28.836102 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:28.836111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:28.836207 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:28.836350 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:28.837745 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.838481 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.839945 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:28.840389 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:28.840490 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:28.843309 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:28.843333 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:28.843350 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:28.843391 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:28.846151 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:28.846784 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:28.846791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:28.846793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:28.846923 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:28.847240 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:28.847244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:28.847246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:28.847590 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:28.847799 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:28.848067 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 20:18:28.848225 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:18:28.849197 18813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:28.884383 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.883646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44397 (local address 127.0.0.1:47472)
0923 20:18:28.883969 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.884125 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260923 20:18:28.885565 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:28.885146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44397 (local address 127.0.0.1:47488)
0923 20:18:28.885384 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:28.885453 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260923 20:18:28.937805 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 18687
I20260923 20:18:28.955605 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 18308
I20260923 20:18:28.956909 18371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:29.072687 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:29.071992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:44397 (local address 127.0.0.1:47504)
0923 20:18:29.072178 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:29.072338 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:44397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 20:18:29.073006 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 18308
I20260923 20:18:29.098295 5453 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 20:18:29.444639 5453 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 20:18:29.444891 5453 mini_ranger.cc:66] Stopping Ranger...
I20260923 20:18:29.790953 5453 mini_ranger.cc:68] Stopped Ranger
2026-09-23 20:18:29.791 UTC [17360] LOG: received smart shutdown request
2026-09-23 20:18:29.794 UTC [17360] LOG: background worker "logical replication launcher" (PID 17369) exited with exit code 1
2026-09-23 20:18:29.794 UTC [17364] LOG: shutting down
2026-09-23 20:18:29.967 UTC [17360] LOG: database system is shut down
2026-09-23T20:18:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52838 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:18:30 dist-test-slave-vl24 krb5kdc[18886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:18:30 dist-test-slave-vl24 krb5kdc[18886](info): set up 2 sockets
Sep 23 20:18:30 dist-test-slave-vl24 krb5kdc[18886](info): commencing operation
krb5kdc: starting...
W20260923 20:18:31.206365 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:18:31 dist-test-slave-vl24 krb5kdc[18886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:18:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:18:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:18:31.456064 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:39487
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:39487,127.5.83.125:44135,127.5.83.124:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:18:31.798297 18899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:31.798658 18899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:31.798841 18899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:31.806483 18899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:18:31.806594 18899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:31.806641 18899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:31.806694 18899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:18:31.806738 18899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:18:31.816886 18899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:39487,127.5.83.125:44135,127.5.83.124:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:39487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:31.818199 18899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:31.819454 18899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:31.831375 18904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:31.831507 18905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:31.831766 18907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:31.832427 18899 server_base.cc:1034] running on GCE node
I20260923 20:18:31.833294 18899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:31.834861 18899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:31.836079 18899 hybrid_clock.cc:648] HybridClock initialized: now 1790194711836036 us; error 38 us; skew 500 ppm
I20260923 20:18:31.836397 18899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:31 dist-test-slave-vl24 krb5kdc[18886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:31.839648 18899 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:18:31.841347 18899 webserver.cc:460] Webserver started at http://127.5.83.126:44283/ using document root <none> and password file <none>
I20260923 20:18:31.842080 18899 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:31.842212 18899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:31.842559 18899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:18:31.845687 18899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "8b9fae1a26dd4afcb97162ace9cb90f6"
format_stamp: "Formatted at 2026-09-23 20:18:31 on dist-test-slave-vl24"
server_key: "1d953fdb4e5d0d5bc17b9e413232fe70"
server_key_iv: "75f59a9e644ef2fedc31cbf1c7fec117"
server_key_version: "encryptionkey@0"
I20260923 20:18:31.846535 18899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "8b9fae1a26dd4afcb97162ace9cb90f6"
format_stamp: "Formatted at 2026-09-23 20:18:31 on dist-test-slave-vl24"
server_key: "1d953fdb4e5d0d5bc17b9e413232fe70"
server_key_iv: "75f59a9e644ef2fedc31cbf1c7fec117"
server_key_version: "encryptionkey@0"
I20260923 20:18:31.851943 18899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:18:31.855700 18914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:31.857342 18899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:18:31.857553 18899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "8b9fae1a26dd4afcb97162ace9cb90f6"
format_stamp: "Formatted at 2026-09-23 20:18:31 on dist-test-slave-vl24"
server_key: "1d953fdb4e5d0d5bc17b9e413232fe70"
server_key_iv: "75f59a9e644ef2fedc31cbf1c7fec117"
server_key_version: "encryptionkey@0"
I20260923 20:18:31.857779 18899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:31.877102 18899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:31.881567 18899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:31.881882 18899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:31.901918 18899 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:39487
I20260923 20:18:31.901960 18965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:39487 every 8 connection(s)
I20260923 20:18:31.903406 18899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:18:31.907311 18966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:31.910804 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 18899
I20260923 20:18:31.911033 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:18:31.911876 5453 external_mini_cluster.cc:1428] Setting key 37bf15f164772771eb51b46b1818d45a
I20260923 20:18:31.916298 18966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:18:31 dist-test-slave-vl24 krb5kdc[18886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.125@KRBTEST.COM" created.
I20260923 20:18:31.940128 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:31.920339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.5.83.126:39487)
0923 20:18:31.922225 (+ 1886us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:31.922233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:31.922284 (+ 51us) server_negotiation.cc:407] Connection header received
0923 20:18:31.923812 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:31.923818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:31.924054 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:31.924260 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:31.925538 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:31.926700 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:31.928143 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:31.928779 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:31.928974 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:31.932640 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:31.932648 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:31.932662 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:31.932697 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:31.935572 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:31.936348 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:31.936359 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:31.936363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:31.936565 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:31.936986 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:31.936991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:31.936995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:31.937501 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:31.937748 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:31.938325 (+ 577us) server_negotiation.cc:299] Negotiation successful
0923 20:18:31.938631 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1672,"spinlock_wait_cycles":49920,"thread_start_us":1467,"threads_started":1}
I20260923 20:18:31.940148 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:31.920339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:39487 (local address 127.0.0.1:56704)
0923 20:18:31.921350 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:31.921404 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:31.923450 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:31.924270 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:31.924276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:31.925013 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:31.925344 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:31.925360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:31.926867 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:31.926872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:31.927997 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:31.928009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:31.929009 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:31.929015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:31.929195 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:31.929918 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:31.929940 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:31.932389 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:31.935726 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:31.935737 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:31.935749 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:31.936213 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:31.936682 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:31.936686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:31.936690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:31.936870 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:31.937652 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:31.937660 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:31.937915 (+ 255us) client_negotiation.cc:769] Sending connection context
0923 20:18:31.938167 (+ 252us) client_negotiation.cc:240] Negotiation successful
0923 20:18:31.938769 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":137,"thread_start_us":533,"threads_started":1}
I20260923 20:18:31.943400 18966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:18:31.944566 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:31.944128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32786)
0923 20:18:31.944298 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:31.944414 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 20:18:31.944849 18916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
I20260923 20:18:31.946650 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:31.946144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32802)
0923 20:18:31.946387 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:31.946476 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260923 20:18:31.946885 18966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:44135: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.125.keytab.
Entry for principal kudu/127.5.83.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.125.keytab.
I20260923 20:18:31.968389 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.125:44135
--webserver_interface=127.5.83.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.125.keytab
--principal=kudu/127.5.83.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:39487,127.5.83.125:44135,127.5.83.124:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:18:31.996378 18966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } attempt: 1
I20260923 20:18:31.997727 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:31.997259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32812)
0923 20:18:31.997482 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:31.997576 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260923 20:18:31.998860 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:31.998466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32818)
0923 20:18:31.998624 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:31.998706 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 20:18:31.999059 18966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:44135: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
I20260923 20:18:32.067337 18966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } attempt: 2
I20260923 20:18:32.068619 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.068057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32820)
0923 20:18:32.068320 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.068413 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 20:18:32.069722 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.069367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32822)
0923 20:18:32.069518 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.069598 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 20:18:32.069971 18966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:44135: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
I20260923 20:18:32.161273 18966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } attempt: 3
I20260923 20:18:32.162571 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.162078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32824)
0923 20:18:32.162304 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.162424 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 20:18:32.163797 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.163367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32836)
0923 20:18:32.163545 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.163634 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260923 20:18:32.164050 18966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:44135: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
I20260923 20:18:32.307335 18966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } attempt: 4
I20260923 20:18:32.308671 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.308077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32846)
0923 20:18:32.308419 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.308514 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260923 20:18:32.309744 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.309372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32852)
0923 20:18:32.309535 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.309626 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 20:18:32.309928 18966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:44135: Network error: Client connection negotiation failed: client connection to 127.5.83.125:44135: connect: Connection refused (error 111)
W20260923 20:18:32.319064 18973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:32.319444 18973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:32.319604 18973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:32.327478 18973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:18:32.327649 18973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:32.327765 18973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:32.327872 18973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:18:32.327958 18973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:18:32.338313 18973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:39487,127.5.83.125:44135,127.5.83.124:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.125.keytab
--principal=kudu/127.5.83.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.125:44135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.83.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:32.339756 18973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:32.341214 18973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:32.353684 18981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:32.353721 18979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:32.354269 18978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:32.354660 18973 server_base.cc:1034] running on GCE node
I20260923 20:18:32.355521 18973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:32.356998 18973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:32.358199 18973 hybrid_clock.cc:648] HybridClock initialized: now 1790194712358133 us; error 54 us; skew 500 ppm
I20260923 20:18:32.358505 18973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:32 dist-test-slave-vl24 krb5kdc[18886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194712, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:32.364998 18973 init.cc:377] Logged in from keytab as kudu/127.5.83.125@KRBTEST.COM (short username kudu)
I20260923 20:18:32.366539 18973 webserver.cc:460] Webserver started at http://127.5.83.125:35475/ using document root <none> and password file <none>
I20260923 20:18:32.367218 18973 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:32.367316 18973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:32.367574 18973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:18:32.370599 18973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/data/instance:
uuid: "a698b11c8955451997fd087ad9afb36e"
format_stamp: "Formatted at 2026-09-23 20:18:32 on dist-test-slave-vl24"
server_key: "d44ed2b97cdfd67e3f1e3fc306dc7187"
server_key_iv: "9416edf8b50e21b6059e248729d11830"
server_key_version: "encryptionkey@0"
I20260923 20:18:32.371397 18973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/wal/instance:
uuid: "a698b11c8955451997fd087ad9afb36e"
format_stamp: "Formatted at 2026-09-23 20:18:32 on dist-test-slave-vl24"
server_key: "d44ed2b97cdfd67e3f1e3fc306dc7187"
server_key_iv: "9416edf8b50e21b6059e248729d11830"
server_key_version: "encryptionkey@0"
I20260923 20:18:32.376784 18973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260923 20:18:32.380285 18988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:32.381783 18973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:18:32.381946 18973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/wal
uuid: "a698b11c8955451997fd087ad9afb36e"
format_stamp: "Formatted at 2026-09-23 20:18:32 on dist-test-slave-vl24"
server_key: "d44ed2b97cdfd67e3f1e3fc306dc7187"
server_key_iv: "9416edf8b50e21b6059e248729d11830"
server_key_version: "encryptionkey@0"
I20260923 20:18:32.382180 18973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:32.412403 18973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:32.417069 18973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:32.417320 18973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:32.437000 18973 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.125:44135
I20260923 20:18:32.437040 19039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.125:44135 every 8 connection(s)
I20260923 20:18:32.438400 18973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
I20260923 20:18:32.442162 19040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:32.443706 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 18973
I20260923 20:18:32.443925 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-1/wal/instance
I20260923 20:18:32.444456 5453 external_mini_cluster.cc:1428] Setting key fe64f89356f5fc54153415e92cf65bad
I20260923 20:18:32.452106 19040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:18:32 dist-test-slave-vl24 krb5kdc[18886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194712, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.125@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.124@KRBTEST.COM" created.
I20260923 20:18:32.476085 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.455272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.5.83.126:39487)
0923 20:18:32.455805 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:32.455812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:32.456845 (+ 1033us) server_negotiation.cc:407] Connection header received
0923 20:18:32.458761 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:32.458771 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:32.458875 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:32.459015 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:32.460520 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.461793 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.463468 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.464205 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.464360 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:32.468759 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:32.468764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:32.468767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:32.468811 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:32.472236 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:32.473193 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:32.473201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:32.473207 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:32.473359 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:32.473831 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:32.473839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:32.473844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:32.474277 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:32.474438 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:32.475307 (+ 869us) server_negotiation.cc:299] Negotiation successful
0923 20:18:32.475572 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":261,"threads_started":1}
I20260923 20:18:32.476927 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.455777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:39487 (local address 127.0.0.1:56720)
0923 20:18:32.456667 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.456714 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:32.458482 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:32.459205 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:32.459214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:32.459748 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:32.460313 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:32.460341 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.461956 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.461960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:32.463310 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:32.463334 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.464475 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.464481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:32.464826 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:32.465652 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:32.465680 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:32.468416 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:32.472417 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:32.472424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:32.472438 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:32.473062 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:32.473483 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:32.473487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:32.473489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:32.473682 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:32.474434 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:32.474441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:32.474825 (+ 384us) client_negotiation.cc:769] Sending connection context
0923 20:18:32.475142 (+ 317us) client_negotiation.cc:240] Negotiation successful
0923 20:18:32.475567 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":350,"threads_started":1}
I20260923 20:18:32.478991 19040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:18:32 dist-test-slave-vl24 krb5kdc[18886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194712, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.125@KRBTEST.COM for kudu/127.5.83.125@KRBTEST.COM
I20260923 20:18:32.499336 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.480011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32862)
0923 20:18:32.480245 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.480266 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:32.480381 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:32.481662 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:32.481666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:32.482073 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:32.482211 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:32.482219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.484862 (+ 2643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.484867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:32.485759 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:32.485770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.487398 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.487403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:32.487595 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:32.488319 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:32.488341 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:32.491095 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:32.496609 (+ 5514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:32.496613 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:32.496616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:32.496991 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:32.497510 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:32.497513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:32.497516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:32.497661 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:32.498456 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:32.498460 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:32.498596 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 20:18:32.498756 (+ 160us) client_negotiation.cc:240] Negotiation successful
0923 20:18:32.499034 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":20}
I20260923 20:18:32.499883 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.480213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32862 (local address 127.5.83.125:44135)
0923 20:18:32.481086 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:32.481099 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:32.481137 (+ 38us) server_negotiation.cc:407] Connection header received
0923 20:18:32.481205 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:32.481218 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:32.481482 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:32.481671 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:32.482382 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.484673 (+ 2291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.485948 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.487225 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.487485 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:32.491379 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:32.491402 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:32.491426 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:32.491479 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:32.496354 (+ 4875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:32.497154 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:32.497168 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:32.497173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:32.497354 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:32.497818 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:32.497823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:32.497825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:32.498313 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:32.498497 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:32.499002 (+ 505us) server_negotiation.cc:299] Negotiation successful
0923 20:18:32.499224 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":637,"thread_start_us":460,"threads_started":1}
I20260923 20:18:32.502455 19040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:18:32.503578 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.503072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52702)
0923 20:18:32.503310 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.503439 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 20:18:32.503834 18992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.124.keytab.
Entry for principal kudu/127.5.83.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.124.keytab.
I20260923 20:18:32.505833 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.505338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52712)
0923 20:18:32.505539 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.505666 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260923 20:18:32.506157 19040 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:42963: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
I20260923 20:18:32.506666 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.124:42963
--webserver_interface=127.5.83.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.124.keytab
--principal=kudu/127.5.83.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:39487,127.5.83.125:44135,127.5.83.124:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:18:32.555557 19040 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } attempt: 1
I20260923 20:18:32.556933 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.556405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52726)
0923 20:18:32.556664 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.556770 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 20:18:32.557910 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.557572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52740)
0923 20:18:32.557737 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.557803 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:18:32.558120 19040 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:42963: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
I20260923 20:18:32.609210 18966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } attempt: 5
Sep 23 20:18:32 dist-test-slave-vl24 krb5kdc[18886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for kudu/127.5.83.125@KRBTEST.COM
I20260923 20:18:32.625159 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.609949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32868)
0923 20:18:32.610219 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.610241 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:32.610370 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:32.610768 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:32.610773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:32.611289 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:32.611527 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:32.611539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.612630 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.612636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:32.614024 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:32.614040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.614884 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.614891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:32.615077 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:32.615782 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:32.615813 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:32.618456 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:32.622619 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:32.622624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:32.622626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:32.623036 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:32.623464 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:32.623467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:32.623470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:32.623603 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:32.624225 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:32.624228 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:32.624401 (+ 173us) client_negotiation.cc:769] Sending connection context
0923 20:18:32.624638 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 20:18:32.624914 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 20:18:32.625305 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.610250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.5.83.125:44135)
0923 20:18:32.610436 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:32.610440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:32.610459 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:18:32.610514 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:32.610518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:32.610594 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:32.610693 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:32.611708 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.612457 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.614184 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.614720 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.614819 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:32.618734 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:32.618743 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:32.618751 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:32.618809 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:32.622431 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:32.623180 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:32.623186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:32.623190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:32.623346 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:32.623709 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:32.623713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:32.623717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:32.624108 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:32.624259 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:32.624756 (+ 497us) server_negotiation.cc:299] Negotiation successful
0923 20:18:32.625003 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:18:32.626330 19040 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } attempt: 2
I20260923 20:18:32.626698 18966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:18:32.627373 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.626894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52752)
0923 20:18:32.627139 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.627244 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260923 20:18:32.627914 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.627394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52758)
0923 20:18:32.627643 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.627735 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260923 20:18:32.628438 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.628051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52770)
0923 20:18:32.628191 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.628292 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 20:18:32.628697 19040 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:42963: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
I20260923 20:18:32.628952 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.628493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52786)
0923 20:18:32.628751 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.628850 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260923 20:18:32.629154 18966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:42963: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
I20260923 20:18:32.680437 18966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } attempt: 1
I20260923 20:18:32.681674 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.681184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52798)
0923 20:18:32.681406 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.681516 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 20:18:32.682710 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.682388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52802)
0923 20:18:32.682539 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.682610 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 20:18:32.683004 18966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:42963: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
I20260923 20:18:32.719957 19040 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } attempt: 3
I20260923 20:18:32.721243 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.720755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52804)
0923 20:18:32.720986 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.721078 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:18:32.722422 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.721980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52820)
0923 20:18:32.722214 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.722302 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260923 20:18:32.722710 19040 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:42963: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
I20260923 20:18:32.751257 18966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } attempt: 2
I20260923 20:18:32.752635 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.752054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52822)
0923 20:18:32.752294 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.752414 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:18:32.753808 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.753424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52830)
0923 20:18:32.753617 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.753692 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 20:18:32.754084 18966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:42963: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
W20260923 20:18:32.851178 19048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:32.851547 19048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:32.851709 19048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:32.859100 19048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:18:32.859267 19048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:32.859386 19048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:32.859491 19048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:18:32.859570 19048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:18:32.860345 18966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } attempt: 3
I20260923 20:18:32.861527 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.861064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52846)
0923 20:18:32.861267 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.861366 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 20:18:32.862615 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.862223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52858)
0923 20:18:32.862445 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.862513 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260923 20:18:32.862840 18966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:42963: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
I20260923 20:18:32.865912 19040 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } attempt: 4
I20260923 20:18:32.866915 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.866499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52870)
0923 20:18:32.866686 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.866769 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 20:18:32.867802 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.867499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52872)
0923 20:18:32.867641 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.867708 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:18:32.868016 19040 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:42963: Network error: Client connection negotiation failed: client connection to 127.5.83.124:42963: connect: Connection refused (error 111)
I20260923 20:18:32.870746 19048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:39487,127.5.83.125:44135,127.5.83.124:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.124.keytab
--principal=kudu/127.5.83.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.124:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.83.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:32.872076 19048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:32.873450 19048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:32.886228 19053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:32.886409 19056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:32.886343 19054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:32.887701 19048 server_base.cc:1034] running on GCE node
I20260923 20:18:32.888497 19048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:32.890158 19048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:32.891412 19048 hybrid_clock.cc:648] HybridClock initialized: now 1790194712891370 us; error 39 us; skew 500 ppm
I20260923 20:18:32.891767 19048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:32 dist-test-slave-vl24 krb5kdc[18886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194712, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:32.895022 19048 init.cc:377] Logged in from keytab as kudu/127.5.83.124@KRBTEST.COM (short username kudu)
I20260923 20:18:32.896473 19048 webserver.cc:460] Webserver started at http://127.5.83.124:36275/ using document root <none> and password file <none>
I20260923 20:18:32.897209 19048 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:32.897354 19048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:32.897665 19048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:18:32.900686 19048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/data/instance:
uuid: "cf2a6fc7722a4f2ca3258e67277a255d"
format_stamp: "Formatted at 2026-09-23 20:18:32 on dist-test-slave-vl24"
server_key: "d07a6f526c33588df2400a8fad706682"
server_key_iv: "2d30cdfc0be878ecfcc5086f90765863"
server_key_version: "encryptionkey@0"
I20260923 20:18:32.901543 19048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/wal/instance:
uuid: "cf2a6fc7722a4f2ca3258e67277a255d"
format_stamp: "Formatted at 2026-09-23 20:18:32 on dist-test-slave-vl24"
server_key: "d07a6f526c33588df2400a8fad706682"
server_key_iv: "2d30cdfc0be878ecfcc5086f90765863"
server_key_version: "encryptionkey@0"
I20260923 20:18:32.906656 19048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260923 20:18:32.910027 19063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:32.911293 19048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:18:32.911496 19048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/wal
uuid: "cf2a6fc7722a4f2ca3258e67277a255d"
format_stamp: "Formatted at 2026-09-23 20:18:32 on dist-test-slave-vl24"
server_key: "d07a6f526c33588df2400a8fad706682"
server_key_iv: "2d30cdfc0be878ecfcc5086f90765863"
server_key_version: "encryptionkey@0"
I20260923 20:18:32.911689 19048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:32.928176 19048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:32.932171 19048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:32.932411 19048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:32.951123 19048 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.124:42963
I20260923 20:18:32.951197 19114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.124:42963 every 8 connection(s)
I20260923 20:18:32.952510 19048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
I20260923 20:18:32.956269 19115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:32.961102 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 19048
I20260923 20:18:32.961285 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790194506409421-5453-0/minicluster-data/master-2/wal/instance
I20260923 20:18:32.961925 5453 external_mini_cluster.cc:1428] Setting key fa504578461972a7d86a20a5875a4ca8
I20260923 20:18:32.963901 19115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:18:32 dist-test-slave-vl24 krb5kdc[18886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194712, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.124@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
Sep 23 20:18:32 dist-test-slave-vl24 krb5kdc[18886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.124@KRBTEST.COM
I20260923 20:18:32.982898 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.966523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.5.83.126:39487)
0923 20:18:32.966713 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:32.966720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:32.967408 (+ 688us) server_negotiation.cc:407] Connection header received
0923 20:18:32.967680 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:32.967688 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:32.967788 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:32.967907 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:32.969184 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.969881 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.971428 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.972002 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.972097 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:32.975928 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:32.975933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:32.975936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:32.975970 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:32.979465 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:32.980305 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:32.980312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:32.980319 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:32.980489 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:32.981044 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:32.981048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:32.981050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:32.981307 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:32.981478 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:32.982322 (+ 844us) server_negotiation.cc:299] Negotiation successful
0923 20:18:32.982593 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:18:32.984004 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.963795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.5.83.124:42963)
0923 20:18:32.964655 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:32.964666 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:32.964707 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:18:32.966516 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:32.966536 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:32.967066 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:32.967443 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:32.969023 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.970332 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.971756 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.972574 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.972685 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:32.977217 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:32.977227 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:32.977244 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:32.977280 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:32.980653 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:32.981477 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:32.981485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:32.981489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:32.981617 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:32.982007 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:32.982010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:32.982012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:32.982413 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:32.982698 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:32.983285 (+ 587us) server_negotiation.cc:299] Negotiation successful
0923 20:18:32.983522 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":627,"thread_start_us":402,"threads_started":1}
I20260923 20:18:32.984055 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.966539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:39487 (local address 127.0.0.1:56736)
0923 20:18:32.967280 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.967322 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:32.967515 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:32.967972 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:32.967977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:32.968567 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:32.969007 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:32.969022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.970034 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.970039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:32.971285 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:32.971297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.972210 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.972217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:32.972634 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:32.973401 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:32.973422 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:32.975729 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:32.979606 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:32.979616 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:32.979625 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:32.980182 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:32.980648 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:32.980652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:32.980654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:32.980921 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:32.981447 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:32.981457 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:32.981891 (+ 434us) client_negotiation.cc:769] Sending connection context
0923 20:18:32.982188 (+ 297us) client_negotiation.cc:240] Negotiation successful
0923 20:18:32.982627 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":71,"thread_start_us":363,"threads_started":1}
I20260923 20:18:32.985743 19115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:18:32 dist-test-slave-vl24 krb5kdc[18886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194712, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.124@KRBTEST.COM for kudu/127.5.83.125@KRBTEST.COM
Sep 23 20:18:32 dist-test-slave-vl24 krb5kdc[18886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:33.001137 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.986590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:44135 (local address 127.0.0.1:32880)
0923 20:18:32.986737 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:32.986757 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:32.986868 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:32.987244 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:32.987249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:32.987736 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:32.987880 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:32.987888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.989274 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.989280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:32.990650 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:32.990662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.991546 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.991549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:32.991708 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:32.992271 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:32.992296 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:32.994642 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:32.998626 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:32.998631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:32.998635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:32.999215 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:32.999609 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:32.999615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:32.999619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:32.999784 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:33.000261 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:33.000268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:33.000453 (+ 185us) client_negotiation.cc:769] Sending connection context
0923 20:18:33.000632 (+ 179us) client_negotiation.cc:240] Negotiation successful
0923 20:18:33.000885 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 20:18:33.001569 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.986514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.5.83.125:44135)
0923 20:18:32.986704 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:32.986709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:32.986839 (+ 130us) server_negotiation.cc:407] Connection header received
0923 20:18:32.987010 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:32.987016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:32.987095 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:32.987197 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:32.988024 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.989136 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.990783 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.991417 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.991554 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:32.994840 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:32.994848 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:32.994853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:32.994901 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:32.998484 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:32.999336 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:32.999340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:32.999342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:32.999489 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:32.999895 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:32.999898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:32.999900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.000144 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:33.000252 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:33.000754 (+ 502us) server_negotiation.cc:299] Negotiation successful
0923 20:18:33.000988 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:18:33.002630 19115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:18:33.003763 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:32.986988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.5.83.126:39487)
0923 20:18:32.987177 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:32.987184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:32.987203 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:18:32.987370 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:32.987375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:32.987461 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:32.987568 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:32.989016 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.990119 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.991745 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:32.992387 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:32.992511 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:32.997442 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:32.997452 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:32.997459 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:32.997510 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:33.000585 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.001411 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.001417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.001420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.001595 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.002117 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.002122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.002125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.002521 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:33.002765 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:33.003156 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 20:18:33.003408 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 23 20:18:33 dist-test-slave-vl24 krb5kdc[18886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194712, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.124@KRBTEST.COM for kudu/127.5.83.124@KRBTEST.COM
Sep 23 20:18:33 dist-test-slave-vl24 krb5kdc[18886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.125@KRBTEST.COM
I20260923 20:18:33.017726 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:33.003520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52888)
0923 20:18:33.003882 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:33.003902 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:33.004003 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:33.004305 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:33.004309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:33.004782 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:33.004950 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:33.004961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.006171 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.006181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:33.007163 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:33.007175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.008005 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.008011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:33.008216 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:33.008907 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:33.008924 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:33.011159 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:33.014738 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:33.014742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:33.014745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:33.015152 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:33.015611 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:33.015615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:33.015619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:33.015796 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:33.016504 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:33.016512 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:33.016723 (+ 211us) client_negotiation.cc:769] Sending connection context
0923 20:18:33.016912 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 20:18:33.017244 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260923 20:18:33.017839 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:33.003738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52888 (local address 127.5.83.124:42963)
0923 20:18:33.003879 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:33.003885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:33.003946 (+ 61us) server_negotiation.cc:407] Connection header received
0923 20:18:33.004128 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:33.004133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:33.004197 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:33.004292 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:33.005111 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.006058 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.007315 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.007857 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.007949 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:33.011393 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:33.011402 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:33.011409 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:33.011458 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:33.014551 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.015296 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.015305 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.015311 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.015488 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.015928 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.015932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.015934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.016360 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:33.016576 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:33.017002 (+ 426us) server_negotiation.cc:299] Negotiation successful
0923 20:18:33.017440 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":171,"server-negotiator.queue_time_us":57}
I20260923 20:18:33.021776 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:33.005682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.5.83.125:44135)
0923 20:18:33.005851 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:33.005857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:33.005899 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:18:33.006115 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:33.006122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:33.006199 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:33.006299 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:33.007615 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.008645 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.010004 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.010423 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.010581 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:33.015444 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:33.015449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:33.015453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:33.015487 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:33.019040 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.019962 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.019966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.019968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.020099 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.020510 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.020513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.020515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.020832 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:33.021037 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:33.021380 (+ 343us) server_negotiation.cc:299] Negotiation successful
0923 20:18:33.021553 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:18:33.024022 19115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d: Bootstrap starting.
I20260923 20:18:33.026964 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:18:33.028196 19115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:33.029584 19115 log.cc:824] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:33.032279 19115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d: No bootstrap required, opened a new log
I20260923 20:18:33.040048 18966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } attempt: 4
I20260923 20:18:33.047061 19115 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } }
I20260923 20:18:33.047735 19115 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:33.047917 19115 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2a6fc7722a4f2ca3258e67277a255d, State: Initialized, Role: FOLLOWER
I20260923 20:18:33.049005 19115 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } }
I20260923 20:18:33.056262 19129 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } } }
I20260923 20:18:33.058984 19129 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:18:33.058842 19115 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [sys.catalog]: configured and running, proceeding with master startup.
Sep 23 20:18:33 dist-test-slave-vl24 krb5kdc[18886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194711, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for kudu/127.5.83.124@KRBTEST.COM
I20260923 20:18:33.072090 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:33.033599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.5.83.125:44135)
0923 20:18:33.033765 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:33.033770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:33.045650 (+ 11880us) server_negotiation.cc:407] Connection header received
0923 20:18:33.046133 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:33.046139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:33.046220 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:33.046283 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:33.048631 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.054724 (+ 6093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.057148 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.057770 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.057889 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:33.061826 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:33.061837 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:33.061845 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:33.062283 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:33.067229 (+ 4946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.068646 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.068656 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.068662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.068891 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.069505 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.069513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.069518 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.070042 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:33.070368 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:33.071175 (+ 807us) server_negotiation.cc:299] Negotiation successful
0923 20:18:33.071528 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:18:33.072974 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:33.040771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52908)
0923 20:18:33.041228 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:33.041255 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:33.041403 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:33.055861 (+ 14458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:33.055866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:33.056398 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:33.056650 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:33.056664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.058718 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.058728 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:33.060217 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:33.060232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.061413 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.061420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:33.061616 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:33.062286 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:33.062312 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:33.065320 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:33.069878 (+ 4558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:33.069886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:33.069891 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:33.070404 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:33.070926 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:33.070932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:33.070935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:33.071093 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:33.071904 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:33.071908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:33.072047 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 20:18:33.072186 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 20:18:33.072514 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"spinlock_wait_cycles":1408}
I20260923 20:18:33.074388 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:33.033785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.5.83.126:39487)
0923 20:18:33.034021 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:33.034029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:33.044684 (+ 10655us) server_negotiation.cc:407] Connection header received
0923 20:18:33.045334 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:33.045343 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:33.045453 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:33.045545 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:33.048351 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.049794 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.056085 (+ 6291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.056877 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.056981 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:33.064305 (+ 7324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:33.064311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:33.064316 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:33.064368 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:33.069141 (+ 4773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.070831 (+ 1690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.070839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.070846 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.070999 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.071652 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.071657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.071660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.072035 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:33.072340 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:33.073803 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0923 20:18:33.074023 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 20:18:33.074458 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:33.041668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.5.83.124:42963)
0923 20:18:33.055374 (+ 13706us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:33.055381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:33.055407 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:18:33.055460 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:33.055464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:33.055537 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:33.056829 (+ 1292us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:33.057148 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.058549 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.060408 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.061228 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.061936 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:33.065548 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:33.065555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:33.065559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:33.065599 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:33.069691 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.070594 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.070601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.070606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.070782 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.071232 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.071240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.071244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.071734 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:33.073639 (+ 1905us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:33.073783 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 20:18:33.074038 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13537,"thread_start_us":13951,"threads_started":1}
I20260923 20:18:33.080329 18966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6: Bootstrap starting.
I20260923 20:18:33.084484 18966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:33.086201 18966 log.cc:824] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:33.087893 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:33.033112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.5.83.124:42963)
0923 20:18:33.033563 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:33.033574 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:33.052894 (+ 19320us) server_negotiation.cc:407] Connection header received
0923 20:18:33.053185 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:33.053194 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:33.053300 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:33.053401 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:33.059386 (+ 5985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.060419 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.065242 (+ 4823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.066024 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.066136 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:33.078586 (+ 12450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:33.078593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:33.078597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:33.078638 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:33.083331 (+ 4693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.084566 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.084575 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.084581 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.084786 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.085389 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.085397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.085402 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.085855 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:33.086988 (+ 1133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:33.087219 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 20:18:33.087477 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260923 20:18:33.090799 18966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6: No bootstrap required, opened a new log
W20260923 20:18:33.091028 19140 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:18:33.091202 19140 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:18:33.087966 19048 master_runner.cc:186] Error getting master registration for 127.5.83.126:39487: OK, instance_id { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" instance_seqno: 1790194711896242 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:18:33.099076 19048 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:18:33.102432 18966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } }
I20260923 20:18:33.102869 18966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:33.103010 18966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9fae1a26dd4afcb97162ace9cb90f6, State: Initialized, Role: FOLLOWER
I20260923 20:18:33.103685 18966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } }
I20260923 20:18:33.104805 19143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } } }
I20260923 20:18:33.105343 19143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:18:33.106244 18966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 20:18:33.123472 19154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:18:33.123735 19154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:18:33.116782 18899 master_runner.cc:186] Error getting master registration for 127.5.83.125:44135: OK, instance_id { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" instance_seqno: 1790194712431387 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:18:33.128968 18899 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 20:18:33.134950 18920 tablet.cc:2369] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:18:33.167266 19040 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } attempt: 5
Sep 23 20:18:33 dist-test-slave-vl24 krb5kdc[18886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194712, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.125@KRBTEST.COM for kudu/127.5.83.124@KRBTEST.COM
W20260923 20:18:33.184628 19069 tablet.cc:2369] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:18:33.189541 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:33.170562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:42963 (local address 127.0.0.1:52914)
0923 20:18:33.171005 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:33.171026 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:33.171152 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:33.171728 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:33.171733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:33.172297 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:33.172559 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:33.172573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.174066 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.174072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:33.175484 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:33.175499 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.176828 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.176834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:33.177027 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:33.177746 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:33.177771 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:33.180642 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:33.184943 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:33.184949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:33.184953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:33.185524 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:33.186040 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:33.186045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:33.186050 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:33.186224 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:33.186976 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:33.186982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:33.187145 (+ 163us) client_negotiation.cc:769] Sending connection context
0923 20:18:33.187295 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 20:18:33.189083 (+ 1788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325}
I20260923 20:18:33.190639 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:33.168709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.5.83.124:42963)
0923 20:18:33.168925 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:33.168932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:33.171264 (+ 2332us) server_negotiation.cc:407] Connection header received
0923 20:18:33.171321 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:33.171326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:33.171419 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:33.171527 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:33.172732 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.173880 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.175655 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:33.176432 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:33.176618 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:33.180889 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:33.180896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:33.180901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:33.180945 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:33.184757 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.185727 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.185733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.185737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.185902 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:33.186354 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:33.186359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:33.186362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:33.186757 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:33.186911 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:33.190023 (+ 3112us) server_negotiation.cc:299] Negotiation successful
0923 20:18:33.190287 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 20:18:33.196823 19040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e: Bootstrap starting.
I20260923 20:18:33.202768 19040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:33.204366 19040 log.cc:824] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:33.208058 19040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e: No bootstrap required, opened a new log
I20260923 20:18:33.221590 19040 raft_consensus.cc:348] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } }
I20260923 20:18:33.222150 19040 raft_consensus.cc:374] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:33.222314 19040 raft_consensus.cc:729] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a698b11c8955451997fd087ad9afb36e, State: Initialized, Role: FOLLOWER
I20260923 20:18:33.223057 19040 consensus_queue.cc:260] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } }
I20260923 20:18:33.224156 19157 sys_catalog.cc:455] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } } }
I20260923 20:18:33.224553 19157 sys_catalog.cc:458] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:18:33.225643 19040 sys_catalog.cc:564] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:18:33.243141 18973 master_runner.cc:263] No leader master found from master a698b11c8955451997fd087ad9afb36e
I20260923 20:18:33.243296 18973 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 20:18:33.246102 19168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:18:33.246241 19168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:18:33.349985 19157 raft_consensus.cc:482] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:18:33.350361 19157 raft_consensus.cc:504] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } }
I20260923 20:18:33.352216 19157 leader_election.cc:288] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b9fae1a26dd4afcb97162ace9cb90f6 (127.5.83.126:39487), cf2a6fc7722a4f2ca3258e67277a255d (127.5.83.124:42963)
I20260923 20:18:33.353482 18941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a698b11c8955451997fd087ad9afb36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" is_pre_election: true
I20260923 20:18:33.354146 18941 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a698b11c8955451997fd087ad9afb36e in term 0.
I20260923 20:18:33.355060 18989 leader_election.cc:302] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9fae1a26dd4afcb97162ace9cb90f6, a698b11c8955451997fd087ad9afb36e; no voters:
I20260923 20:18:33.356340 19090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a698b11c8955451997fd087ad9afb36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" is_pre_election: true
I20260923 20:18:33.357045 19090 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a698b11c8955451997fd087ad9afb36e in term 0.
I20260923 20:18:33.358045 19157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:18:33.358227 19157 raft_consensus.cc:482] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:18:33.358347 19157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:33.367781 19157 raft_consensus.cc:504] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } }
I20260923 20:18:33.371110 18941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a698b11c8955451997fd087ad9afb36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6"
I20260923 20:18:33.371389 18941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:33.375197 18941 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a698b11c8955451997fd087ad9afb36e in term 1.
I20260923 20:18:33.375815 18989 leader_election.cc:302] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9fae1a26dd4afcb97162ace9cb90f6, a698b11c8955451997fd087ad9afb36e; no voters:
I20260923 20:18:33.380285 19090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a698b11c8955451997fd087ad9afb36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2a6fc7722a4f2ca3258e67277a255d"
I20260923 20:18:33.380640 19090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:33.384241 19090 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a698b11c8955451997fd087ad9afb36e in term 1.
I20260923 20:18:33.385368 19157 leader_election.cc:288] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [CANDIDATE]: Term 1 election: Requested vote from peers 8b9fae1a26dd4afcb97162ace9cb90f6 (127.5.83.126:39487), cf2a6fc7722a4f2ca3258e67277a255d (127.5.83.124:42963)
I20260923 20:18:33.385861 19157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:18:33.386389 19157 raft_consensus.cc:686] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [term 1 LEADER]: Becoming Leader. State: Replica: a698b11c8955451997fd087ad9afb36e, State: Running, Role: LEADER
I20260923 20:18:33.387861 19157 consensus_queue.cc:237] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } }
I20260923 20:18:33.414194 19170 sys_catalog.cc:455] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a698b11c8955451997fd087ad9afb36e. Latest consensus state: current_term: 1 leader_uuid: "a698b11c8955451997fd087ad9afb36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } } }
I20260923 20:18:33.414777 19170 sys_catalog.cc:458] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [sys.catalog]: This master's current role is: LEADER
W20260923 20:18:33.420395 18994 tablet.cc:2369] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:18:33.423560 19172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:18:33.426502 19172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:18:33.448563 19090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [term 1 FOLLOWER]: Refusing update from remote peer a698b11c8955451997fd087ad9afb36e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:18:33.449927 19170 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:18:33.457616 18941 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [term 1 FOLLOWER]: Refusing update from remote peer a698b11c8955451997fd087ad9afb36e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:18:33.458710 19170 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:18:33.495991 19143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a698b11c8955451997fd087ad9afb36e. Latest consensus state: current_term: 1 leader_uuid: "a698b11c8955451997fd087ad9afb36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } } }
I20260923 20:18:33.496420 19143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:18:33.505208 19173 mvcc.cc:205] Tried to move back new op lower bound from 7332637546272907264 to 7332637546047926272. Current Snapshot: MvccSnapshot[applied={T|T < 7332637546272907264}]
I20260923 20:18:33.518889 19175 mvcc.cc:205] Tried to move back new op lower bound from 7332637546272907264 to 7332637546047926272. Current Snapshot: MvccSnapshot[applied={T|T < 7332637546272907264}]
I20260923 20:18:33.532369 19172 catalog_manager.cc:1347] Generated new cluster ID: 137acbddf67d4a3a9c49666413199909
I20260923 20:18:33.532500 19172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:18:33.548159 19143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a698b11c8955451997fd087ad9afb36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } } }
I20260923 20:18:33.548504 19143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:18:33.550586 19157 sys_catalog.cc:455] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a698b11c8955451997fd087ad9afb36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } } }
I20260923 20:18:33.550920 19157 sys_catalog.cc:458] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:33.561313 19129 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a698b11c8955451997fd087ad9afb36e. Latest consensus state: current_term: 1 leader_uuid: "a698b11c8955451997fd087ad9afb36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } } }
I20260923 20:18:33.561638 19129 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:18:33.568981 19157 sys_catalog.cc:455] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a698b11c8955451997fd087ad9afb36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } } }
I20260923 20:18:33.569465 19157 sys_catalog.cc:458] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:33.582166 19129 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a698b11c8955451997fd087ad9afb36e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9fae1a26dd4afcb97162ace9cb90f6" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39487 } } peers { permanent_uuid: "a698b11c8955451997fd087ad9afb36e" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 44135 } } peers { permanent_uuid: "cf2a6fc7722a4f2ca3258e67277a255d" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 42963 } } }
I20260923 20:18:33.582475 19129 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:18:33.601840 19172 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:18:33.603189 19172 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:18:33.615087 19172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a698b11c8955451997fd087ad9afb36e: Generated new TSK 0
I20260923 20:18:33.617396 19172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:18:34.093354 19140 catalog_manager.cc:1259] Loaded cluster ID: 137acbddf67d4a3a9c49666413199909
I20260923 20:18:34.093458 19140 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d: loading cluster ID for follower catalog manager: success
I20260923 20:18:34.096014 19140 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d: acquiring CA information for follower catalog manager: success
I20260923 20:18:34.097815 19140 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cf2a6fc7722a4f2ca3258e67277a255d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:18:34.126081 19154 catalog_manager.cc:1259] Loaded cluster ID: 137acbddf67d4a3a9c49666413199909
I20260923 20:18:34.126206 19154 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6: loading cluster ID for follower catalog manager: success
I20260923 20:18:34.129408 19154 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6: acquiring CA information for follower catalog manager: success
I20260923 20:18:34.131582 19154 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b9fae1a26dd4afcb97162ace9cb90f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:18:34.180173 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 18899
I20260923 20:18:34.181520 18961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:34.291343 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 18899
I20260923 20:18:34.313825 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 18973
I20260923 20:18:34.315135 19035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:34.425678 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 18973
I20260923 20:18:34.449057 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 19048
I20260923 20:18:34.450245 19110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:34.558527 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 19048
2026-09-23T20:18:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4461 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:18:34 dist-test-slave-vl24 krb5kdc[19191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:18:34 dist-test-slave-vl24 krb5kdc[19191](info): set up 2 sockets
Sep 23 20:18:34 dist-test-slave-vl24 krb5kdc[19191](info): commencing operation
krb5kdc: starting...
W20260923 20:18:35.702829 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:18:35 dist-test-slave-vl24 krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:18:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:18:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:18:35.954339 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:43765
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:43765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:18:36.292420 19204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:36.292824 19204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:36.293004 19204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:36.300581 19204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:18:36.300737 19204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:36.300843 19204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:36.300949 19204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:18:36.301045 19204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:18:36.311223 19204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:43765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:43765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:36.312645 19204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:36.314011 19204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:36.326483 19210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:36.326568 19209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:36.326485 19212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:36.327549 19204 server_base.cc:1034] running on GCE node
I20260923 20:18:36.328333 19204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:36.329802 19204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:36.331060 19204 hybrid_clock.cc:648] HybridClock initialized: now 1790194716331006 us; error 35 us; skew 500 ppm
I20260923 20:18:36.331424 19204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:36 dist-test-slave-vl24 krb5kdc[19191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194716, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:36.334667 19204 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:18:36.336174 19204 webserver.cc:460] Webserver started at http://127.5.83.126:36095/ using document root <none> and password file <none>
I20260923 20:18:36.336966 19204 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:36.337113 19204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:36.337461 19204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:18:36.340444 19204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "20fa3d5623fe4eeba9954b1914296b4b"
format_stamp: "Formatted at 2026-09-23 20:18:36 on dist-test-slave-vl24"
server_key: "b647480e4ebab3b73c769cd98310f862"
server_key_iv: "ca363e1f423493d83ab7eb31af12827a"
server_key_version: "encryptionkey@0"
I20260923 20:18:36.341270 19204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "20fa3d5623fe4eeba9954b1914296b4b"
format_stamp: "Formatted at 2026-09-23 20:18:36 on dist-test-slave-vl24"
server_key: "b647480e4ebab3b73c769cd98310f862"
server_key_iv: "ca363e1f423493d83ab7eb31af12827a"
server_key_version: "encryptionkey@0"
I20260923 20:18:36.346511 19204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:18:36.349880 19219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:36.351207 19204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:18:36.351408 19204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "20fa3d5623fe4eeba9954b1914296b4b"
format_stamp: "Formatted at 2026-09-23 20:18:36 on dist-test-slave-vl24"
server_key: "b647480e4ebab3b73c769cd98310f862"
server_key_iv: "ca363e1f423493d83ab7eb31af12827a"
server_key_version: "encryptionkey@0"
I20260923 20:18:36.351612 19204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:36.373595 19204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:36.378629 19204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:36.378983 19204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:36.400398 19204 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:43765
I20260923 20:18:36.400410 19270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:43765 every 8 connection(s)
I20260923 20:18:36.401921 19204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:18:36.405747 19271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:36.409106 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 19204
I20260923 20:18:36.409363 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:18:36.410070 5453 external_mini_cluster.cc:1428] Setting key 9c6d62246490999d165cb6f3a93ad248
I20260923 20:18:36.419191 19271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b: Bootstrap starting.
I20260923 20:18:36.422744 19271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:36.423779 19271 log.cc:824] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:18:36 dist-test-slave-vl24 krb5kdc[19191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:36.427050 19271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b: No bootstrap required, opened a new log
I20260923 20:18:36.435854 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:36.412145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.5.83.126:43765)
0923 20:18:36.412996 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:36.413011 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:36.413062 (+ 51us) server_negotiation.cc:407] Connection header received
0923 20:18:36.414941 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:36.414986 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:36.415529 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:36.415985 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:36.417481 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:36.419424 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:36.420876 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:36.421921 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:36.422139 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:36.426539 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:36.426551 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:36.426570 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:36.426619 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:36.430738 (+ 4119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:36.431604 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:36.431618 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:36.431628 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:36.431875 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:36.432316 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:36.432320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:36.432323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:36.433138 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:36.433456 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:36.433925 (+ 469us) server_negotiation.cc:299] Negotiation successful
0923 20:18:36.434408 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":350,"threads_started":1}
I20260923 20:18:36.438108 19271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20fa3d5623fe4eeba9954b1914296b4b" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43765 } }
I20260923 20:18:36.438627 19271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:36.438751 19271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20fa3d5623fe4eeba9954b1914296b4b, State: Initialized, Role: FOLLOWER
I20260923 20:18:36.439385 19271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20fa3d5623fe4eeba9954b1914296b4b" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43765 } }
I20260923 20:18:36.439652 19271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:18:36.439806 19271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:18:36.439986 19271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:36.440062 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:18:36.443199 19271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20fa3d5623fe4eeba9954b1914296b4b" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43765 } }
I20260923 20:18:36.443717 19271 leader_election.cc:302] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20fa3d5623fe4eeba9954b1914296b4b; no voters:
I20260923 20:18:36.444356 19271 leader_election.cc:288] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:18:36.444491 19276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:18:36.445231 19276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [term 1 LEADER]: Becoming Leader. State: Replica: 20fa3d5623fe4eeba9954b1914296b4b, State: Running, Role: LEADER
I20260923 20:18:36.445838 19276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20fa3d5623fe4eeba9954b1914296b4b" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43765 } }
I20260923 20:18:36.446590 19271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:18:36.449023 19277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20fa3d5623fe4eeba9954b1914296b4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20fa3d5623fe4eeba9954b1914296b4b" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43765 } } }
I20260923 20:18:36.449563 19277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:36.449095 19278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20fa3d5623fe4eeba9954b1914296b4b. Latest consensus state: current_term: 1 leader_uuid: "20fa3d5623fe4eeba9954b1914296b4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20fa3d5623fe4eeba9954b1914296b4b" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43765 } } }
I20260923 20:18:36.450168 19278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b [sys.catalog]: This master's current role is: LEADER
W20260923 20:18:36.456694 19292 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:18:36.456835 19292 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:18:36.456957 19285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:18:36.458652 19285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:18:36.466341 19285 catalog_manager.cc:1347] Generated new cluster ID: 9d93cc3b0ae64c78b73307dbddd8ef26
I20260923 20:18:36.466459 19285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:18:36.475363 19285 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:18:36.476760 19285 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:18:36.492429 19285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20fa3d5623fe4eeba9954b1914296b4b: Generated new TSK 0
I20260923 20:18:36.493387 19285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:18:36.501761 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 19204
I20260923 20:18:36.503127 19266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:36.595908 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 19204
2026-09-23T20:18:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2031 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:18:36 dist-test-slave-vl24 krb5kdc[19299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:18:36 dist-test-slave-vl24 krb5kdc[19299](info): set up 2 sockets
Sep 23 20:18:36 dist-test-slave-vl24 krb5kdc[19299](info): commencing operation
krb5kdc: starting...
W20260923 20:18:37.702288 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:18:37 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:18:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:18:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:18:37.956593 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:41617
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:41617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:18:38.292029 19312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:38.292306 19312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:38.292485 19312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:38.299710 19312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:18:38.299821 19312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:38.299932 19312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:38.300038 19312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:18:38.300117 19312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:18:38.310434 19312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:41617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:41617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:38.311808 19312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:38.313198 19312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:38.325495 19317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:38.325577 19320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:38.325547 19318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:38.327322 19312 server_base.cc:1034] running on GCE node
I20260923 20:18:38.328223 19312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:38.329849 19312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:38.331086 19312 hybrid_clock.cc:648] HybridClock initialized: now 1790194718331036 us; error 41 us; skew 500 ppm
I20260923 20:18:38.331453 19312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:38 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:38.338565 19312 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:18:38.340384 19312 webserver.cc:460] Webserver started at http://127.5.83.126:37499/ using document root <none> and password file <none>
I20260923 20:18:38.341171 19312 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:38.341325 19312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:38.341645 19312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:18:38.344816 19312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "f975b77b24884acf9e62723bc0508af5"
format_stamp: "Formatted at 2026-09-23 20:18:38 on dist-test-slave-vl24"
server_key: "ededf893e6becdac5127c737dea63ee1"
server_key_iv: "653d62c2edd4a42d090249e8fd230f24"
server_key_version: "encryptionkey@0"
I20260923 20:18:38.345611 19312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "f975b77b24884acf9e62723bc0508af5"
format_stamp: "Formatted at 2026-09-23 20:18:38 on dist-test-slave-vl24"
server_key: "ededf893e6becdac5127c737dea63ee1"
server_key_iv: "653d62c2edd4a42d090249e8fd230f24"
server_key_version: "encryptionkey@0"
I20260923 20:18:38.351137 19312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:18:38.354743 19327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:38.356176 19312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:18:38.356410 19312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "f975b77b24884acf9e62723bc0508af5"
format_stamp: "Formatted at 2026-09-23 20:18:38 on dist-test-slave-vl24"
server_key: "ededf893e6becdac5127c737dea63ee1"
server_key_iv: "653d62c2edd4a42d090249e8fd230f24"
server_key_version: "encryptionkey@0"
I20260923 20:18:38.356727 19312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:38.388422 19312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:38.393021 19312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:38.393392 19312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:38.414726 19312 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:41617
I20260923 20:18:38.414749 19378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:41617 every 8 connection(s)
I20260923 20:18:38.416299 19312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:18:38.420204 19379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:38.421484 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 19312
I20260923 20:18:38.421708 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:18:38.422555 5453 external_mini_cluster.cc:1428] Setting key c7c7d2b9cc94e7867b0ded1df48c14cb
I20260923 20:18:38.433485 19379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Bootstrap starting.
I20260923 20:18:38.436478 19379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Neither blocks nor log segments found. Creating new log.
Sep 23 20:18:38 dist-test-slave-vl24 krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:38.437789 19379 log.cc:824] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:38.440848 19379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: No bootstrap required, opened a new log
I20260923 20:18:38.446909 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:38.424788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.5.83.126:41617)
0923 20:18:38.425600 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:38.425613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:38.425660 (+ 47us) server_negotiation.cc:407] Connection header received
0923 20:18:38.427431 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:38.427453 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:38.427873 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:38.428292 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:38.429860 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:38.431294 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:38.432831 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:38.433683 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:38.433902 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:38.438891 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:38.438909 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:38.438931 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:38.438996 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:38.442410 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:38.443243 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:38.443254 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:38.443272 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:38.443490 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:38.443916 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:38.443920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:38.443922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:38.444620 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:38.444898 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:38.445335 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 20:18:38.445761 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":591,"thread_start_us":362,"threads_started":1}
I20260923 20:18:38.449922 19379 raft_consensus.cc:348] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } }
I20260923 20:18:38.450284 19379 raft_consensus.cc:374] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:38.450378 19379 raft_consensus.cc:729] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f975b77b24884acf9e62723bc0508af5, State: Initialized, Role: FOLLOWER
I20260923 20:18:38.450927 19379 consensus_queue.cc:260] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } }
I20260923 20:18:38.451148 19379 raft_consensus.cc:388] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:18:38.451303 19379 raft_consensus.cc:482] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:18:38.451472 19379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:38.453688 19379 raft_consensus.cc:504] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } }
I20260923 20:18:38.454160 19379 leader_election.cc:302] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f975b77b24884acf9e62723bc0508af5; no voters:
I20260923 20:18:38.454880 19379 leader_election.cc:288] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:18:38.454998 19384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:18:38.455823 19384 raft_consensus.cc:686] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 1 LEADER]: Becoming Leader. State: Replica: f975b77b24884acf9e62723bc0508af5, State: Running, Role: LEADER
I20260923 20:18:38.456439 19384 consensus_queue.cc:237] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } }
I20260923 20:18:38.457342 19379 sys_catalog.cc:564] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:18:38.459585 19386 sys_catalog.cc:455] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f975b77b24884acf9e62723bc0508af5. Latest consensus state: current_term: 1 leader_uuid: "f975b77b24884acf9e62723bc0508af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } } }
I20260923 20:18:38.459694 19385 sys_catalog.cc:455] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f975b77b24884acf9e62723bc0508af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } } }
I20260923 20:18:38.460019 19386 sys_catalog.cc:458] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:38.460119 19385 sys_catalog.cc:458] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:38.461591 19389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:18:38.469378 19389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:18:38.481244 19389 catalog_manager.cc:1347] Generated new cluster ID: c2e37df7b9ac43e9b8206ca59f4fff9a
I20260923 20:18:38.481382 19389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:18:38.497219 19389 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:18:38.499136 19389 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:18:38.517625 19389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Generated new TSK 0
I20260923 20:18:38.518666 19389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:18:38.581442 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41617
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:18:38.912941 19405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:38.913229 19405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:38.913321 19405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:38.920696 19405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:38.920796 19405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:38.920909 19405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:18:38.930542 19405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:38.931864 19405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:38.933209 19405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:38.946872 19411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:38.947335 19410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:38.947744 19405 server_base.cc:1034] running on GCE node
W20260923 20:18:38.949082 19413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:38.949965 19405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:38.951305 19405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:38.952605 19405 hybrid_clock.cc:648] HybridClock initialized: now 1790194718952483 us; error 95 us; skew 500 ppm
I20260923 20:18:38.952888 19405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:38 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:38.956127 19405 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:18:38.957775 19405 webserver.cc:460] Webserver started at http://127.5.83.65:37605/ using document root <none> and password file <none>
I20260923 20:18:38.958599 19405 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:38.958717 19405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:38.959185 19405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:18:38.962445 19405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "6f06b98f9bc2418e8decbbd3b059ee30"
format_stamp: "Formatted at 2026-09-23 20:18:38 on dist-test-slave-vl24"
server_key: "5cb64f1427ae89acb5b223ee5ed7164e"
server_key_iv: "32f65bfa77f1e73a9c8f1a05d92fbfc8"
server_key_version: "encryptionkey@0"
I20260923 20:18:38.963135 19405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "6f06b98f9bc2418e8decbbd3b059ee30"
format_stamp: "Formatted at 2026-09-23 20:18:38 on dist-test-slave-vl24"
server_key: "5cb64f1427ae89acb5b223ee5ed7164e"
server_key_iv: "32f65bfa77f1e73a9c8f1a05d92fbfc8"
server_key_version: "encryptionkey@0"
I20260923 20:18:38.968358 19405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 20:18:38.971761 19420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:38.973315 19405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:18:38.973467 19405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "6f06b98f9bc2418e8decbbd3b059ee30"
format_stamp: "Formatted at 2026-09-23 20:18:38 on dist-test-slave-vl24"
server_key: "5cb64f1427ae89acb5b223ee5ed7164e"
server_key_iv: "32f65bfa77f1e73a9c8f1a05d92fbfc8"
server_key_version: "encryptionkey@0"
I20260923 20:18:38.973631 19405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:39.009212 19405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:39.013535 19405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:39.013784 19405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:39.014765 19405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:39.016151 19405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:39.016227 19405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:39.016319 19405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:39.016362 19405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:39.050460 19405 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:43983
I20260923 20:18:39.050500 19532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:43983 every 8 connection(s)
I20260923 20:18:39.051781 19405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:18:39.056972 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 19405
I20260923 20:18:39.057197 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:18:39.058032 5453 external_mini_cluster.cc:1428] Setting key 769c653e0d84a3869f9809c474fd3c64
Sep 23 20:18:39 dist-test-slave-vl24 krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:39.074190 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:39.054284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:47809 (local address 127.5.83.126:41617)
0923 20:18:39.054749 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:39.054754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:39.055694 (+ 940us) server_negotiation.cc:407] Connection header received
0923 20:18:39.058255 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:39.058260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:39.058397 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:39.058492 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:39.060066 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:39.060982 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:39.062694 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:39.063107 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:39.063249 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:39.067260 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:39.067267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:39.067270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:39.067312 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:39.070109 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:39.071078 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:39.071082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:39.071084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:39.071203 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:39.071757 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:39.071761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:39.071763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:39.072096 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:39.072271 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:39.073431 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0923 20:18:39.073711 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":211,"threads_started":1}
I20260923 20:18:39.075292 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:39.054842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:47809)
0923 20:18:39.055532 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:39.055569 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:39.057925 (+ 2356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:39.058771 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:39.058779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:39.059270 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:39.059854 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:39.059872 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:39.061159 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:39.061169 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:39.062527 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:39.062539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:39.063361 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:39.063369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:39.063597 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:39.064314 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:39.064357 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:39.066887 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:39.070265 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:39.070278 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:39.070298 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:39.070953 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:39.071321 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:39.071329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:39.071362 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:39.071610 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:39.072299 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:39.072309 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:39.072868 (+ 559us) client_negotiation.cc:769] Sending connection context
0923 20:18:39.073224 (+ 356us) client_negotiation.cc:240] Negotiation successful
0923 20:18:39.073804 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":277,"threads_started":1}
I20260923 20:18:39.077134 19533 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41617
I20260923 20:18:39.077507 19533 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:39.078482 19533 heartbeater.cc:511] Master 127.5.83.126:41617 requested a full tablet report, sending...
I20260923 20:18:39.081110 19344 ts_manager.cc:194] Registered new tserver with Master: 6f06b98f9bc2418e8decbbd3b059ee30 (127.5.83.65:43983)
I20260923 20:18:39.082633 19344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:47809
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:18:39.116300 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41617
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:18:39.457171 19539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:39.457499 19539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:39.457712 19539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:39.465411 19539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:39.465525 19539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:39.465730 19539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:18:39.476082 19539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:39.477599 19539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:39.478960 19539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:39.492425 19547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:39.492511 19545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:39.492594 19544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:39.493254 19539 server_base.cc:1034] running on GCE node
I20260923 20:18:39.493927 19539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:39.495440 19539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:39.496896 19539 hybrid_clock.cc:648] HybridClock initialized: now 1790194719496805 us; error 86 us; skew 500 ppm
I20260923 20:18:39.497228 19539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:39 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194719, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:39.500591 19539 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:18:39.502135 19539 webserver.cc:460] Webserver started at http://127.5.83.66:43179/ using document root <none> and password file <none>
I20260923 20:18:39.502875 19539 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:39.503001 19539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:39.503315 19539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:18:39.506394 19539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "b3b0e97b6b744100b2c791fa691a56f2"
format_stamp: "Formatted at 2026-09-23 20:18:39 on dist-test-slave-vl24"
server_key: "42c6bd75fba72b8ce038611a85ac05e3"
server_key_iv: "9cae2fc5e8de0a3f7b802521f82ef63c"
server_key_version: "encryptionkey@0"
I20260923 20:18:39.507246 19539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "b3b0e97b6b744100b2c791fa691a56f2"
format_stamp: "Formatted at 2026-09-23 20:18:39 on dist-test-slave-vl24"
server_key: "42c6bd75fba72b8ce038611a85ac05e3"
server_key_iv: "9cae2fc5e8de0a3f7b802521f82ef63c"
server_key_version: "encryptionkey@0"
I20260923 20:18:39.513072 19539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:18:39.517763 19554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:39.519405 19539 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 20:18:39.519604 19539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "b3b0e97b6b744100b2c791fa691a56f2"
format_stamp: "Formatted at 2026-09-23 20:18:39 on dist-test-slave-vl24"
server_key: "42c6bd75fba72b8ce038611a85ac05e3"
server_key_iv: "9cae2fc5e8de0a3f7b802521f82ef63c"
server_key_version: "encryptionkey@0"
I20260923 20:18:39.519815 19539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:39.538664 19539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:39.542858 19539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:39.543154 19539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:39.544173 19539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:39.545686 19539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:39.545797 19539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:39.545943 19539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:39.546021 19539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:39.580781 19539 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:38437
I20260923 20:18:39.580822 19666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:38437 every 8 connection(s)
I20260923 20:18:39.582080 19539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:18:39.591763 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 19539
I20260923 20:18:39.591990 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:18:39.592567 5453 external_mini_cluster.cc:1428] Setting key 68ec975fd18d01a6ca124b30af862fc9
Sep 23 20:18:39 dist-test-slave-vl24 krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194719, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:39.602707 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:39.584789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:49629 (local address 127.5.83.126:41617)
0923 20:18:39.585211 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:39.585216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:39.586169 (+ 953us) server_negotiation.cc:407] Connection header received
0923 20:18:39.587980 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:39.587987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:39.588109 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:39.588226 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:39.589756 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:39.590483 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:39.592048 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:39.592763 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:39.592907 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:39.596785 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:39.596790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:39.596794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:39.596825 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:39.599402 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:39.600328 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:39.600332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:39.600334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:39.600469 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:39.601011 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:39.601015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:39.601016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:39.601269 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:39.601378 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:39.602270 (+ 892us) server_negotiation.cc:299] Negotiation successful
0923 20:18:39.602454 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":204,"threads_started":1}
I20260923 20:18:39.604157 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:39.585334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:49629)
0923 20:18:39.586006 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:39.586048 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:39.587701 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:39.588452 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:39.588460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:39.589027 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:39.589589 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:39.589605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:39.590631 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:39.590638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:39.591875 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:39.591886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:39.592965 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:39.592971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:39.593163 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:39.593833 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:39.593858 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:39.596560 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:39.599555 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:39.599566 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:39.599585 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:39.600205 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:39.600626 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:39.600634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:39.600640 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:39.600895 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:39.601397 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:39.601408 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:39.601871 (+ 463us) client_negotiation.cc:769] Sending connection context
0923 20:18:39.602174 (+ 303us) client_negotiation.cc:240] Negotiation successful
0923 20:18:39.602770 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":257,"threads_started":1}
I20260923 20:18:39.605739 19667 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41617
I20260923 20:18:39.606040 19667 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:39.606761 19667 heartbeater.cc:511] Master 127.5.83.126:41617 requested a full tablet report, sending...
I20260923 20:18:39.608359 19344 ts_manager.cc:194] Registered new tserver with Master: b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437)
I20260923 20:18:39.609642 19344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:49629
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:18:39.650161 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41617
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:18:39.985562 19673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:39.985853 19673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:39.985957 19673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:39.993139 19673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:39.993247 19673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:39.993360 19673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:18:40.003104 19673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:40.004397 19673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:40.005977 19673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:40.020052 19679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:40.020165 19678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:40.020191 19681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:40.021131 19673 server_base.cc:1034] running on GCE node
I20260923 20:18:40.021751 19673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:40.022747 19673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:40.024012 19673 hybrid_clock.cc:648] HybridClock initialized: now 1790194720023959 us; error 34 us; skew 500 ppm
I20260923 20:18:40.024348 19673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:40 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194720, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:40.027535 19673 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:18:40.029122 19673 webserver.cc:460] Webserver started at http://127.5.83.67:38497/ using document root <none> and password file <none>
I20260923 20:18:40.029820 19673 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:40.029968 19673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:40.030287 19673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:18:40.033334 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "dd26eabc60604ad3910de89dd711c379"
format_stamp: "Formatted at 2026-09-23 20:18:40 on dist-test-slave-vl24"
server_key: "8ffb28903d5c0fb5fddf416c4537b056"
server_key_iv: "e768d24382061484dc24faf4ad2c5540"
server_key_version: "encryptionkey@0"
I20260923 20:18:40.034140 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "dd26eabc60604ad3910de89dd711c379"
format_stamp: "Formatted at 2026-09-23 20:18:40 on dist-test-slave-vl24"
server_key: "8ffb28903d5c0fb5fddf416c4537b056"
server_key_iv: "e768d24382061484dc24faf4ad2c5540"
server_key_version: "encryptionkey@0"
I20260923 20:18:40.039429 19673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:18:40.042841 19688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:40.044235 19673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:18:40.044412 19673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "dd26eabc60604ad3910de89dd711c379"
format_stamp: "Formatted at 2026-09-23 20:18:40 on dist-test-slave-vl24"
server_key: "8ffb28903d5c0fb5fddf416c4537b056"
server_key_iv: "e768d24382061484dc24faf4ad2c5540"
server_key_version: "encryptionkey@0"
I20260923 20:18:40.044703 19673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:40.060103 19673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:40.064155 19673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:40.064462 19673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:40.065470 19673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:40.066908 19673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:40.067023 19673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:40.067175 19673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:40.067250 19673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:40.086404 19533 heartbeater.cc:503] Master 127.5.83.126:41617 was elected leader, sending a full tablet report...
I20260923 20:18:40.104177 19673 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:45505
I20260923 20:18:40.104270 19800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:45505 every 8 connection(s)
I20260923 20:18:40.105531 19673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:18:40.114641 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 19673
I20260923 20:18:40.114838 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:18:40.115305 5453 external_mini_cluster.cc:1428] Setting key a5d102ba1776259fd7f56b466f1d9a7c
Sep 23 20:18:40 dist-test-slave-vl24 krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194720, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:40.125483 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.107973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:41037 (local address 127.5.83.126:41617)
0923 20:18:40.108450 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:40.108457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:40.109418 (+ 961us) server_negotiation.cc:407] Connection header received
0923 20:18:40.111270 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:40.111275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:40.111362 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:40.111460 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:40.112945 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.113728 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.115341 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.115928 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.116066 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.119806 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:40.119810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:40.119813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:40.119845 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:40.122356 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:40.123068 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:40.123072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:40.123074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:40.123209 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:40.123782 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:40.123785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:40.123787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:40.124049 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:40.124164 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:40.125022 (+ 858us) server_negotiation.cc:299] Negotiation successful
0923 20:18:40.125192 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":208,"threads_started":1}
I20260923 20:18:40.126454 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.108472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:41037)
0923 20:18:40.109238 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.109280 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:40.111010 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:40.111667 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:40.111675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:40.112163 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:40.112775 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:40.112794 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.113869 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.113873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:40.115193 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:40.115205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.116145 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.116150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:40.116352 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.117220 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:40.117251 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:40.119617 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:40.122504 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:40.122511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:40.122523 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:40.122955 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:40.123456 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:40.123460 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:40.123463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:40.123671 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:40.124168 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:40.124174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:40.124670 (+ 496us) client_negotiation.cc:769] Sending connection context
0923 20:18:40.124926 (+ 256us) client_negotiation.cc:240] Negotiation successful
0923 20:18:40.125318 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":264,"threads_started":1}
I20260923 20:18:40.127832 19801 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41617
I20260923 20:18:40.128099 19801 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:40.128895 19801 heartbeater.cc:511] Master 127.5.83.126:41617 requested a full tablet report, sending...
I20260923 20:18:40.130486 19344 ts_manager.cc:194] Registered new tserver with Master: dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505)
I20260923 20:18:40.131106 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:18:40.131580 19344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:41037
I20260923 20:18:40.149034 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.136334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.5.83.126:41617)
0923 20:18:40.136604 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:40.136613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:40.136958 (+ 345us) server_negotiation.cc:407] Connection header received
0923 20:18:40.137087 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:40.137091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:40.137153 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:40.137234 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:40.138548 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.139255 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.140599 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.141013 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.141115 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.143640 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:40.143644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:40.143647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:40.143679 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:40.146321 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:40.147011 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:40.147015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:40.147017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:40.147141 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:40.147523 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:40.147526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:40.147528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:40.147875 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:40.148120 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:40.148437 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 20:18:40.148698 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:18:40.153584 19344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:18:40.161636 19344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:18:40.194519 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.174861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34296)
0923 20:18:40.184754 (+ 9893us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.184797 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:40.185008 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:40.186560 (+ 1552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:40.186568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:40.186603 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:40.187212 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:40.187234 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.189186 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.189190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:40.191163 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:40.191184 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.193526 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.193534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:40.193737 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.193788 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 20:18:40.193932 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 20:18:40.194140 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9605,"thread_start_us":9494,"threads_started":1}
I20260923 20:18:40.194541 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.175287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.5.83.65:43983)
0923 20:18:40.185884 (+ 10597us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:40.185893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:40.185925 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:18:40.186027 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:40.186035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:40.186313 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:40.186533 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:40.187484 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.189026 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.191357 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.193007 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.193138 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.193408 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:40.194007 (+ 599us) server_negotiation.cc:299] Negotiation successful
0923 20:18:40.194190 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10414,"thread_start_us":10248,"threads_started":1}
I20260923 20:18:40.195798 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.178325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:38437 (local address 127.0.0.1:52978)
0923 20:18:40.187248 (+ 8923us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.187269 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:40.187389 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:40.188164 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:40.188173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:40.188196 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:40.188688 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:40.188702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.190569 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.190572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:40.193095 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:40.193109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.194956 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.194961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:40.195082 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.195154 (+ 72us) client_negotiation.cc:769] Sending connection context
0923 20:18:40.195246 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 20:18:40.195396 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8780,"mutex_wait_us":111,"thread_start_us":8069,"threads_started":1}
I20260923 20:18:40.195870 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.178969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.5.83.66:38437)
0923 20:18:40.186204 (+ 7235us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:40.186212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:40.187386 (+ 1174us) server_negotiation.cc:407] Connection header received
0923 20:18:40.187551 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:40.187557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:40.187708 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:40.187919 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:40.188882 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.190401 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.193274 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.194527 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.194663 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.194843 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:40.195349 (+ 506us) server_negotiation.cc:299] Negotiation successful
0923 20:18:40.195530 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7100,"thread_start_us":6964,"threads_started":1}
I20260923 20:18:40.195995 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.175289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:45505 (local address 127.0.0.1:56812)
0923 20:18:40.186743 (+ 11454us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.186769 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:40.187161 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:40.187844 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:40.187850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:40.187888 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:40.188320 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:40.188338 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.190490 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.190498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:40.192582 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:40.192595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.195096 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.195103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:40.195344 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.195377 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 20:18:40.195461 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 20:18:40.195588 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11272,"mutex_wait_us":50,"thread_start_us":11620,"threads_started":1}
I20260923 20:18:40.196847 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.175500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.5.83.67:45505)
0923 20:18:40.184323 (+ 8823us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:40.184335 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:40.186935 (+ 2600us) server_negotiation.cc:407] Connection header received
0923 20:18:40.187376 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:40.187384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:40.187625 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:40.187838 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:40.188609 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.190338 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.193147 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.194583 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.194726 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.194949 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:40.196112 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0923 20:18:40.196464 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8616,"thread_start_us":8426,"threads_started":1}
I20260923 20:18:40.199579 19736 tablet_service.cc:1467] Processing CreateTablet for tablet 12149250ba5543339a990bc225763b3f (DEFAULT_TABLE table=test-table [id=b7417dc8b6f74363a9312db2b65113af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:18:40.199448 19468 tablet_service.cc:1467] Processing CreateTablet for tablet 12149250ba5543339a990bc225763b3f (DEFAULT_TABLE table=test-table [id=b7417dc8b6f74363a9312db2b65113af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:18:40.199644 19602 tablet_service.cc:1467] Processing CreateTablet for tablet 12149250ba5543339a990bc225763b3f (DEFAULT_TABLE table=test-table [id=b7417dc8b6f74363a9312db2b65113af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:18:40.200862 19736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12149250ba5543339a990bc225763b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:40.200906 19602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12149250ba5543339a990bc225763b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:40.200913 19468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12149250ba5543339a990bc225763b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:40.213300 19820 tablet_bootstrap.cc:492] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379: Bootstrap starting.
I20260923 20:18:40.215283 19821 tablet_bootstrap.cc:492] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30: Bootstrap starting.
I20260923 20:18:40.217258 19822 tablet_bootstrap.cc:492] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2: Bootstrap starting.
I20260923 20:18:40.218605 19820 tablet_bootstrap.cc:654] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:40.218758 19821 tablet_bootstrap.cc:654] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:40.219955 19821 log.cc:824] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:40.219957 19820 log.cc:824] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:40.220155 19822 tablet_bootstrap.cc:654] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:40.221421 19822 log.cc:824] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:40.223251 19821 tablet_bootstrap.cc:492] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30: No bootstrap required, opened a new log
I20260923 20:18:40.223264 19820 tablet_bootstrap.cc:492] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379: No bootstrap required, opened a new log
I20260923 20:18:40.223558 19821 ts_tablet_manager.cc:1397] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260923 20:18:40.223546 19820 ts_tablet_manager.cc:1397] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260923 20:18:40.224143 19822 tablet_bootstrap.cc:492] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2: No bootstrap required, opened a new log
I20260923 20:18:40.224429 19822 ts_tablet_manager.cc:1397] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260923 20:18:40.231585 19822 raft_consensus.cc:348] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } } peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } }
I20260923 20:18:40.231971 19822 raft_consensus.cc:374] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:40.232084 19822 raft_consensus.cc:729] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b0e97b6b744100b2c791fa691a56f2, State: Initialized, Role: FOLLOWER
I20260923 20:18:40.232769 19822 consensus_queue.cc:260] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } } peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } }
I20260923 20:18:40.234007 19667 heartbeater.cc:503] Master 127.5.83.126:41617 was elected leader, sending a full tablet report...
I20260923 20:18:40.234714 19822 ts_tablet_manager.cc:1428] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260923 20:18:40.234522 19820 raft_consensus.cc:348] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } } peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } }
I20260923 20:18:40.234711 19821 raft_consensus.cc:348] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } } peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } }
I20260923 20:18:40.235041 19820 raft_consensus.cc:374] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:40.235066 19821 raft_consensus.cc:374] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:40.235147 19821 raft_consensus.cc:729] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f06b98f9bc2418e8decbbd3b059ee30, State: Initialized, Role: FOLLOWER
I20260923 20:18:40.235147 19820 raft_consensus.cc:729] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd26eabc60604ad3910de89dd711c379, State: Initialized, Role: FOLLOWER
I20260923 20:18:40.235776 19820 consensus_queue.cc:260] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } } peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } }
I20260923 20:18:40.235916 19821 consensus_queue.cc:260] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } } peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } }
I20260923 20:18:40.238080 19821 ts_tablet_manager.cc:1428] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260923 20:18:40.238930 19801 heartbeater.cc:503] Master 127.5.83.126:41617 was elected leader, sending a full tablet report...
I20260923 20:18:40.239490 19820 ts_tablet_manager.cc:1428] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
W20260923 20:18:40.304316 19534 tablet.cc:2369] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:18:40.334203 19668 tablet.cc:2369] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:18:40.351737 19827 raft_consensus.cc:482] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:18:40.352108 19827 raft_consensus.cc:504] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } } peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } }
I20260923 20:18:40.353868 19827 leader_election.cc:288] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f06b98f9bc2418e8decbbd3b059ee30 (127.5.83.65:43983), b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437)
W20260923 20:18:40.357129 19802 tablet.cc:2369] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:18:40.361187 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.354445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:60927 (local address 127.5.83.66:38437)
0923 20:18:40.354645 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:40.354651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:40.355344 (+ 693us) server_negotiation.cc:407] Connection header received
0923 20:18:40.355478 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:40.355483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:40.355558 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:40.355665 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:40.356252 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.357137 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.359279 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.360096 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.360267 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.360369 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:40.360776 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 20:18:40.360911 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:18:40.361222 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.354264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.5.83.67:37057)
0923 20:18:40.354615 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.354639 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:40.354880 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:40.355366 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:40.355373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:40.355415 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:40.355629 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:40.355643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.356935 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.356940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:40.358765 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:40.358781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.360266 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.360274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:40.360520 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.360636 (+ 116us) client_negotiation.cc:769] Sending connection context
0923 20:18:40.360752 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 20:18:40.360920 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":125}
I20260923 20:18:40.361305 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.354386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:37057 (local address 127.5.83.65:43983)
0923 20:18:40.354600 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:40.354607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:40.354759 (+ 152us) server_negotiation.cc:407] Connection header received
0923 20:18:40.355068 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:40.355073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:40.355159 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:40.355279 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:40.355777 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.356771 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.358957 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.360044 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.360150 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.360271 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:40.360928 (+ 657us) server_negotiation.cc:299] Negotiation successful
0923 20:18:40.361074 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:18:40.361372 19829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.354367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:38437 (local address 127.5.83.67:60927)
0923 20:18:40.355196 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.355216 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:40.355336 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:40.355682 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:40.355685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:40.355726 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:40.356098 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:40.356109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.357309 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.357317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:40.359133 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:40.359147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.360344 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.360348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:40.360517 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.360633 (+ 116us) client_negotiation.cc:769] Sending connection context
0923 20:18:40.360737 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:18:40.360912 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":33,"thread_start_us":458,"threads_started":1}
I20260923 20:18:40.362218 19488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12149250ba5543339a990bc225763b3f" candidate_uuid: "dd26eabc60604ad3910de89dd711c379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" is_pre_election: true
I20260923 20:18:40.362675 19622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12149250ba5543339a990bc225763b3f" candidate_uuid: "dd26eabc60604ad3910de89dd711c379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b0e97b6b744100b2c791fa691a56f2" is_pre_election: true
I20260923 20:18:40.362874 19488 raft_consensus.cc:2454] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd26eabc60604ad3910de89dd711c379 in term 0.
I20260923 20:18:40.363219 19622 raft_consensus.cc:2454] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd26eabc60604ad3910de89dd711c379 in term 0.
I20260923 20:18:40.363627 19689 leader_election.cc:302] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f06b98f9bc2418e8decbbd3b059ee30, dd26eabc60604ad3910de89dd711c379; no voters:
I20260923 20:18:40.364005 19827 raft_consensus.cc:2781] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:18:40.364171 19827 raft_consensus.cc:482] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:18:40.364279 19827 raft_consensus.cc:3037] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:40.366659 19827 raft_consensus.cc:504] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } } peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } }
I20260923 20:18:40.367280 19827 leader_election.cc:288] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [CANDIDATE]: Term 1 election: Requested vote from peers 6f06b98f9bc2418e8decbbd3b059ee30 (127.5.83.65:43983), b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437)
I20260923 20:18:40.367710 19488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12149250ba5543339a990bc225763b3f" candidate_uuid: "dd26eabc60604ad3910de89dd711c379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f06b98f9bc2418e8decbbd3b059ee30"
I20260923 20:18:40.367789 19622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12149250ba5543339a990bc225763b3f" candidate_uuid: "dd26eabc60604ad3910de89dd711c379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b0e97b6b744100b2c791fa691a56f2"
I20260923 20:18:40.367908 19488 raft_consensus.cc:3037] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:40.368029 19622 raft_consensus.cc:3037] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:40.370186 19488 raft_consensus.cc:2454] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd26eabc60604ad3910de89dd711c379 in term 1.
I20260923 20:18:40.370307 19622 raft_consensus.cc:2454] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd26eabc60604ad3910de89dd711c379 in term 1.
I20260923 20:18:40.370667 19689 leader_election.cc:302] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f06b98f9bc2418e8decbbd3b059ee30, dd26eabc60604ad3910de89dd711c379; no voters:
I20260923 20:18:40.370947 19827 raft_consensus.cc:2781] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:18:40.371353 19827 raft_consensus.cc:686] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 1 LEADER]: Becoming Leader. State: Replica: dd26eabc60604ad3910de89dd711c379, State: Running, Role: LEADER
I20260923 20:18:40.371832 19827 consensus_queue.cc:237] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } } peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } }
I20260923 20:18:40.377540 19344 catalog_manager.cc:5495] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd26eabc60604ad3910de89dd711c379 (127.5.83.67). New cstate: current_term: 1 leader_uuid: "dd26eabc60604ad3910de89dd711c379" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } health_report { overall_health: UNKNOWN } } }
I20260923 20:18:40.437048 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.430763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.5.83.67:45505)
0923 20:18:40.430995 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:40.431001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:40.431106 (+ 105us) server_negotiation.cc:407] Connection header received
0923 20:18:40.431377 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:40.431381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:40.431465 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:40.431675 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:18:40.432100 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.433050 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.434683 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.435149 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.435316 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.435851 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:18:40.436385 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:18:40.436601 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:40.436675 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 20:18:40.436760 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:18:40.445586 19622 raft_consensus.cc:1261] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2 [term 1 FOLLOWER]: Refusing update from remote peer dd26eabc60604ad3910de89dd711c379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:18:40.445587 19488 raft_consensus.cc:1261] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Refusing update from remote peer dd26eabc60604ad3910de89dd711c379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:18:40.446609 19830 consensus_queue.cc:1035] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:18:40.447011 19827 consensus_queue.cc:1035] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:18:40.456234 19833 mvcc.cc:205] Tried to move back new op lower bound from 7332637574934577152 to 7332637574647070720. Current Snapshot: MvccSnapshot[applied={T|T < 7332637574934577152}]
I20260923 20:18:40.458072 19834 mvcc.cc:205] Tried to move back new op lower bound from 7332637574934577152 to 7332637574647070720. Current Snapshot: MvccSnapshot[applied={T|T < 7332637574934577152}]
I20260923 20:18:40.476248 19344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:18:40.476570 19344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:18:40.480288 19344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Sending DeleteTablet for 3 replicas of tablet 12149250ba5543339a990bc225763b3f
I20260923 20:18:40.481544 19468 tablet_service.cc:1514] Processing DeleteTablet for tablet 12149250ba5543339a990bc225763b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:18:40 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:34296
I20260923 20:18:40.481618 19736 tablet_service.cc:1514] Processing DeleteTablet for tablet 12149250ba5543339a990bc225763b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:18:40 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:56812
I20260923 20:18:40.482318 19602 tablet_service.cc:1514] Processing DeleteTablet for tablet 12149250ba5543339a990bc225763b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:18:40 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:52978
I20260923 20:18:40.482386 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 19405
I20260923 20:18:40.492666 19843 tablet_replica.cc:330] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379: stopping tablet replica
I20260923 20:18:40.494199 19843 raft_consensus.cc:2229] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:18:40.495826 19843 raft_consensus.cc:2258] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:18:40.498276 19844 tablet_replica.cc:330] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2: stopping tablet replica
I20260923 20:18:40.498862 19844 raft_consensus.cc:2229] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:18:40.499492 19844 raft_consensus.cc:2258] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:18:40.499711 19843 ts_tablet_manager.cc:1905] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:18:40.500119 19842 tablet_replica.cc:330] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30: stopping tablet replica
I20260923 20:18:40.502533 19844 ts_tablet_manager.cc:1905] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:18:40.507983 19843 ts_tablet_manager.cc:1918] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 23 20:18:40 dist-test-slave-vl24 krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:18:40.508268 19843 log.cc:1196] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/12149250ba5543339a990bc225763b3f
I20260923 20:18:40.509017 19843 ts_tablet_manager.cc:1939] T 12149250ba5543339a990bc225763b3f P dd26eabc60604ad3910de89dd711c379: Deleting consensus metadata
I20260923 20:18:40.510125 19844 ts_tablet_manager.cc:1918] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:18:40.510268 19844 log.cc:1196] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/12149250ba5543339a990bc225763b3f
I20260923 20:18:40.510775 19844 ts_tablet_manager.cc:1939] T 12149250ba5543339a990bc225763b3f P b3b0e97b6b744100b2c791fa691a56f2: Deleting consensus metadata
I20260923 20:18:40.511669 19329 catalog_manager.cc:4841] TS dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505): tablet 12149250ba5543339a990bc225763b3f (table test-table [id=b7417dc8b6f74363a9312db2b65113af]) successfully deleted
I20260923 20:18:40.512343 19842 raft_consensus.cc:2229] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:18:40.512552 19331 catalog_manager.cc:4841] TS b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437): tablet 12149250ba5543339a990bc225763b3f (table test-table [id=b7417dc8b6f74363a9312db2b65113af]) successfully deleted
I20260923 20:18:40.512938 19842 raft_consensus.cc:2258] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:18:40.515023 19842 ts_tablet_manager.cc:1905] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:18:40.517077 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.483799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.5.83.65:43983)
0923 20:18:40.484042 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:40.484052 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:40.496084 (+ 12032us) server_negotiation.cc:407] Connection header received
0923 20:18:40.496733 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:40.496742 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:40.496850 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:40.496936 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:40.499318 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.500681 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.502751 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.503433 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.503545 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.509411 (+ 5866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:40.509432 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:40.509454 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:40.509506 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:40.514005 (+ 4499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:40.514719 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:40.514735 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:40.514738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:40.514879 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:40.515319 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:40.515324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:40.515329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:40.515667 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:40.515994 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:40.516308 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 20:18:40.516567 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 20:18:40.517800 19528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:40.611753 19842 ts_tablet_manager.cc:1918] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:18:40.611982 19842 log.cc:1196] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/12149250ba5543339a990bc225763b3f
I20260923 20:18:40.612591 19842 ts_tablet_manager.cc:1939] T 12149250ba5543339a990bc225763b3f P 6f06b98f9bc2418e8decbbd3b059ee30: Deleting consensus metadata
I20260923 20:18:40.612775 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 19405
I20260923 20:18:40.631767 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 19539
W20260923 20:18:40.631949 19328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.65:43983 (error 108)
I20260923 20:18:40.634617 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.634107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34300)
0923 20:18:40.634314 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.634429 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 20:18:40.634891 19328 catalog_manager.cc:4584] TS 6f06b98f9bc2418e8decbbd3b059ee30 (127.5.83.65:43983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12149250ba5543339a990bc225763b3f: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Sep 23 20:18:40 dist-test-slave-vl24 krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:18:40.648224 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.632768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.5.83.66:38437)
0923 20:18:40.633003 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:40.633011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:40.633038 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:18:40.633145 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:40.633152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:40.633252 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:40.633361 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:40.635231 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.635949 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.637523 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.637927 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.638057 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.642282 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:40.642295 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:40.642311 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:40.642345 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:40.645379 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:40.646089 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:40.646096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:40.646099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:40.646218 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:40.646600 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:40.646605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:40.646607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:40.646985 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:40.647283 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:40.647603 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 20:18:40.647893 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:18:40.649075 19662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:40.688812 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.688205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34302)
0923 20:18:40.688466 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.688607 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260923 20:18:40.689975 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.689557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34316)
0923 20:18:40.689718 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.689828 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 20:18:40.741056 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 19539
I20260923 20:18:40.750679 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.750083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34322)
0923 20:18:40.750351 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.750469 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 20:18:40.751885 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.751451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34334)
0923 20:18:40.751640 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.751729 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 20:18:40.759964 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 19673
Sep 23 20:18:40 dist-test-slave-vl24 krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:18:40.777683 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.760842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.5.83.67:45505)
0923 20:18:40.761034 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:40.761040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:40.761065 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:18:40.761236 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:40.761242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:40.761352 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:40.761493 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:40.762740 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.764015 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.765577 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:40.766302 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:40.766452 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:40.771402 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:40.771414 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:40.771430 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:40.771472 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:40.774664 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:40.775504 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:40.775511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:40.775513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:40.775662 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:40.776105 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:40.776108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:40.776110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:40.776460 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:40.776852 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:40.777153 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 20:18:40.777328 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 20:18:40.778498 19796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:40.832852 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.832235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34350)
0923 20:18:40.832486 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.832665 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260923 20:18:40.833964 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:40.833535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34354)
0923 20:18:40.833761 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:40.833860 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260923 20:18:40.905544 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 19673
I20260923 20:18:40.926368 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 19312
I20260923 20:18:40.927579 19374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:41.042500 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:41.041859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34364)
0923 20:18:41.042157 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:41.042254 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260923 20:18:41.042613 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 19312
I20260923 20:18:41.065382 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:41617
--webserver_interface=127.5.83.126
--webserver_port=37499
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:41617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:18:41.409916 19851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:41.410215 19851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:41.410403 19851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:41.418079 19851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:18:41.418253 19851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:41.418375 19851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:41.418474 19851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:18:41.418560 19851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:18:41.428786 19851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:41617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:41617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=37499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:41.430217 19851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:41.431547 19851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:41.455288 19857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:41.455387 19860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:41.455323 19858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:41.456108 19851 server_base.cc:1034] running on GCE node
I20260923 20:18:41.457135 19851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:41.458787 19851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:41.460048 19851 hybrid_clock.cc:648] HybridClock initialized: now 1790194721459975 us; error 49 us; skew 500 ppm
I20260923 20:18:41.460518 19851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:41 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194721, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:41.464353 19851 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:18:41.466173 19851 webserver.cc:460] Webserver started at http://127.5.83.126:37499/ using document root <none> and password file <none>
I20260923 20:18:41.466930 19851 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:41.467083 19851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:41.473297 19851 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 20:18:41.476364 19867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:41.477864 19851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:18:41.478060 19851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "f975b77b24884acf9e62723bc0508af5"
format_stamp: "Formatted at 2026-09-23 20:18:38 on dist-test-slave-vl24"
server_key: "ededf893e6becdac5127c737dea63ee1"
server_key_iv: "653d62c2edd4a42d090249e8fd230f24"
server_key_version: "encryptionkey@0"
I20260923 20:18:41.479187 19851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:41.506681 19851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:41.511262 19851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:41.511643 19851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:41.531845 19851 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:41617
I20260923 20:18:41.531880 19918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:41617 every 8 connection(s)
I20260923 20:18:41.533483 19851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:18:41.540706 19919 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:18:41.541167 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 19851
I20260923 20:18:41.542006 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:43983
--local_ip_for_outbound_sockets=127.5.83.65
--tserver_master_addrs=127.5.83.126:41617
--webserver_port=37605
--webserver_interface=127.5.83.65
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:18:41.544240 19919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Bootstrap starting.
I20260923 20:18:41.564728 19919 log.cc:824] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:41.577113 19919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:18:41.577661 19919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Bootstrap complete.
I20260923 20:18:41.587917 19919 raft_consensus.cc:348] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } }
I20260923 20:18:41.588901 19919 raft_consensus.cc:729] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f975b77b24884acf9e62723bc0508af5, State: Initialized, Role: FOLLOWER
I20260923 20:18:41.589524 19919 consensus_queue.cc:260] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } }
I20260923 20:18:41.589794 19919 raft_consensus.cc:388] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:18:41.589937 19919 raft_consensus.cc:482] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:18:41.590137 19919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:18:41.594230 19919 raft_consensus.cc:504] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } }
I20260923 20:18:41.594801 19919 leader_election.cc:302] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f975b77b24884acf9e62723bc0508af5; no voters:
I20260923 20:18:41.595516 19919 leader_election.cc:288] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:18:41.595669 19923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:18:41.596982 19923 raft_consensus.cc:686] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 2 LEADER]: Becoming Leader. State: Replica: f975b77b24884acf9e62723bc0508af5, State: Running, Role: LEADER
I20260923 20:18:41.597673 19923 consensus_queue.cc:237] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } }
I20260923 20:18:41.598074 19919 sys_catalog.cc:564] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:18:41.600556 19925 sys_catalog.cc:455] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f975b77b24884acf9e62723bc0508af5. Latest consensus state: current_term: 2 leader_uuid: "f975b77b24884acf9e62723bc0508af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } } }
I20260923 20:18:41.600641 19924 sys_catalog.cc:455] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f975b77b24884acf9e62723bc0508af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } } }
I20260923 20:18:41.601140 19924 sys_catalog.cc:458] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:41.601410 19925 sys_catalog.cc:458] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:41.618386 19928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:18:41.629089 19928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:18:41.631217 19928 catalog_manager.cc:1259] Loaded cluster ID: c2e37df7b9ac43e9b8206ca59f4fff9a
I20260923 20:18:41.631441 19928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:18:41.635991 19928 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 20:18:41.662765 19928 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 20:18:41.685319 19941 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:18:41.686817 19928 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:18:41.688271 19928 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Loaded TSK: 0
I20260923 20:18:41.689757 19928 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:18:41.925107 19921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:41.925387 19921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:41.925541 19921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:41.933023 19921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:41.933139 19921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:41.933346 19921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:18:41.944232 19921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:43983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=37605
--tserver_master_addrs=127.5.83.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:41.945685 19921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:41.947054 19921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:41.962352 19947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:41.962450 19946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:41.962450 19949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:41.964063 19921 server_base.cc:1034] running on GCE node
I20260923 20:18:41.964865 19921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:41.966007 19921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:41.967267 19921 hybrid_clock.cc:648] HybridClock initialized: now 1790194721967211 us; error 45 us; skew 500 ppm
I20260923 20:18:41.967646 19921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:41 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194721, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:41.971174 19921 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:18:41.972959 19921 webserver.cc:460] Webserver started at http://127.5.83.65:37605/ using document root <none> and password file <none>
I20260923 20:18:41.973701 19921 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:41.973865 19921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:41.979552 19921 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:18:41.982534 19956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:41.984174 19921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:18:41.984349 19921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "6f06b98f9bc2418e8decbbd3b059ee30"
format_stamp: "Formatted at 2026-09-23 20:18:38 on dist-test-slave-vl24"
server_key: "5cb64f1427ae89acb5b223ee5ed7164e"
server_key_iv: "32f65bfa77f1e73a9c8f1a05d92fbfc8"
server_key_version: "encryptionkey@0"
I20260923 20:18:41.985410 19921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:42.007028 19921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:42.011368 19921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:42.011656 19921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:42.012812 19921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:42.014389 19921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:42.014526 19921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:42.014673 19921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:42.014763 19921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:42.050262 19921 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:43983
I20260923 20:18:42.050384 20068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:43983 every 8 connection(s)
I20260923 20:18:42.051730 19921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:18:42.057996 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 19921
I20260923 20:18:42.058899 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:38437
--local_ip_for_outbound_sockets=127.5.83.66
--tserver_master_addrs=127.5.83.126:41617
--webserver_port=43179
--webserver_interface=127.5.83.66
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:18:42 dist-test-slave-vl24 krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194721, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:42.077860 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:42.054618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:35685 (local address 127.5.83.126:41617)
0923 20:18:42.055198 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:42.055209 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:42.055886 (+ 677us) server_negotiation.cc:407] Connection header received
0923 20:18:42.058236 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:42.058255 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:42.058721 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:42.059172 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:42.060699 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:42.062833 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:42.064397 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:42.065585 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:42.065810 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:42.069934 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:42.069942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:42.069957 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:42.069998 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:42.073030 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:42.074141 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:42.074148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:42.074153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:42.074323 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:42.074893 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:42.074896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:42.074898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:42.075347 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:42.075496 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:42.076493 (+ 997us) server_negotiation.cc:299] Negotiation successful
0923 20:18:42.076892 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":252,"threads_started":1}
I20260923 20:18:42.078473 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:42.054842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:35685)
0923 20:18:42.055718 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:42.055770 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:42.057591 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:42.059388 (+ 1797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:42.059395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:42.059890 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:42.060404 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:42.060422 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:42.063012 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:42.063016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:42.064237 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:42.064249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:42.065811 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:42.065816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:42.066018 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:42.066738 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:42.066766 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:42.069651 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:42.073229 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:42.073244 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:42.073277 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:42.074001 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:42.074482 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:42.074490 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:42.074497 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:42.074770 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:42.075526 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:42.075538 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:42.075993 (+ 455us) client_negotiation.cc:769] Sending connection context
0923 20:18:42.076343 (+ 350us) client_negotiation.cc:240] Negotiation successful
0923 20:18:42.077027 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":378,"threads_started":1}
I20260923 20:18:42.081329 20069 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41617
I20260923 20:18:42.081635 20069 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:42.082333 20069 heartbeater.cc:511] Master 127.5.83.126:41617 requested a full tablet report, sending...
I20260923 20:18:42.085222 19884 ts_manager.cc:194] Registered new tserver with Master: 6f06b98f9bc2418e8decbbd3b059ee30 (127.5.83.65:43983)
I20260923 20:18:42.087484 19884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:35685
W20260923 20:18:42.408918 20073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:42.409222 20073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:42.409320 20073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:42.417176 20073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:42.417304 20073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:42.417420 20073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:18:42.427376 20073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:38437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=43179
--tserver_master_addrs=127.5.83.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:42.428715 20073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:42.430035 20073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:42.443454 20079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:42.443531 20081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:42.443493 20078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:42.444669 20073 server_base.cc:1034] running on GCE node
I20260923 20:18:42.445369 20073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:42.446338 20073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:42.447562 20073 hybrid_clock.cc:648] HybridClock initialized: now 1790194722447513 us; error 62 us; skew 500 ppm
I20260923 20:18:42.447896 20073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:42 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194722, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:42.450989 20073 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:18:42.452486 20073 webserver.cc:460] Webserver started at http://127.5.83.66:43179/ using document root <none> and password file <none>
I20260923 20:18:42.453248 20073 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:42.453389 20073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:42.458847 20073 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 20:18:42.461727 20088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:42.463111 20073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:18:42.463294 20073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "b3b0e97b6b744100b2c791fa691a56f2"
format_stamp: "Formatted at 2026-09-23 20:18:39 on dist-test-slave-vl24"
server_key: "42c6bd75fba72b8ce038611a85ac05e3"
server_key_iv: "9cae2fc5e8de0a3f7b802521f82ef63c"
server_key_version: "encryptionkey@0"
I20260923 20:18:42.464232 20073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:42.485272 20073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:42.489449 20073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:42.489727 20073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:42.490769 20073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:42.492110 20073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:42.492209 20073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:42.492370 20073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:42.492455 20073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:42.526175 20073 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:38437
I20260923 20:18:42.526263 20200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:38437 every 8 connection(s)
I20260923 20:18:42.527635 20073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:18:42.537706 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 20073
I20260923 20:18:42.538440 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:45505
--local_ip_for_outbound_sockets=127.5.83.67
--tserver_master_addrs=127.5.83.126:41617
--webserver_port=38497
--webserver_interface=127.5.83.67
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:18:42 dist-test-slave-vl24 krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194722, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:42.548223 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:42.530243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:39655 (local address 127.5.83.126:41617)
0923 20:18:42.530429 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:42.530433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:42.531569 (+ 1136us) server_negotiation.cc:407] Connection header received
0923 20:18:42.533506 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:42.533512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:42.533664 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:42.533800 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:42.535241 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:42.535956 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:42.537576 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:42.538130 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:42.538270 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:42.542097 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:42.542103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:42.542106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:42.542139 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:42.544916 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:42.545651 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:42.545658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:42.545666 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:42.545796 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:42.546234 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:42.546243 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:42.546246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:42.546540 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:42.546709 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:42.547589 (+ 880us) server_negotiation.cc:299] Negotiation successful
0923 20:18:42.547829 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:18:42.549540 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:42.530660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:39655)
0923 20:18:42.531420 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:42.531457 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:42.533207 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:42.534044 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:42.534051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:42.534527 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:42.535079 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:42.535096 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:42.536110 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:42.536115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:42.537424 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:42.537435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:42.538369 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:42.538376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:42.538711 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:42.539421 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:42.539448 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:42.541856 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:42.545066 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:42.545075 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:42.545094 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:42.545519 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:42.545915 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:42.545920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:42.545923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:42.546111 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:42.546676 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:42.546684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:42.547106 (+ 422us) client_negotiation.cc:769] Sending connection context
0923 20:18:42.547436 (+ 330us) client_negotiation.cc:240] Negotiation successful
0923 20:18:42.548043 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":293,"threads_started":1}
I20260923 20:18:42.551131 20201 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41617
I20260923 20:18:42.551450 20201 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:42.552304 20201 heartbeater.cc:511] Master 127.5.83.126:41617 requested a full tablet report, sending...
I20260923 20:18:42.553998 19884 ts_manager.cc:194] Registered new tserver with Master: b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437)
I20260923 20:18:42.555385 19884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:39655
W20260923 20:18:42.876709 20204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:42.877002 20204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:42.877095 20204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:42.884565 20204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:42.884684 20204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:42.884799 20204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:18:42.895251 20204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:45505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=38497
--tserver_master_addrs=127.5.83.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:42.896580 20204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:42.897850 20204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:42.910874 20209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:42.910920 20210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:42.910878 20212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:42.912120 20204 server_base.cc:1034] running on GCE node
I20260923 20:18:42.912806 20204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:42.913811 20204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:42.915040 20204 hybrid_clock.cc:648] HybridClock initialized: now 1790194722914976 us; error 68 us; skew 500 ppm
I20260923 20:18:42.915364 20204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:42 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194722, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:42.918486 20204 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:18:42.919888 20204 webserver.cc:460] Webserver started at http://127.5.83.67:38497/ using document root <none> and password file <none>
I20260923 20:18:42.920576 20204 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:42.920706 20204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:42.926025 20204 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260923 20:18:42.928853 20219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:42.930292 20204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:18:42.930480 20204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "dd26eabc60604ad3910de89dd711c379"
format_stamp: "Formatted at 2026-09-23 20:18:40 on dist-test-slave-vl24"
server_key: "8ffb28903d5c0fb5fddf416c4537b056"
server_key_iv: "e768d24382061484dc24faf4ad2c5540"
server_key_version: "encryptionkey@0"
I20260923 20:18:42.931469 20204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:42.953226 20204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:42.957620 20204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:42.957921 20204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:42.958989 20204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:42.960368 20204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:42.960494 20204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:42.960656 20204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:42.960749 20204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:42.994792 20204 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:45505
I20260923 20:18:42.994858 20331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:45505 every 8 connection(s)
I20260923 20:18:42.996235 20204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:18:42.997532 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 20204
Sep 23 20:18:43 dist-test-slave-vl24 krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194722, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:18:43.015141 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:42.998596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.5.83.126:41617)
0923 20:18:42.998817 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:42.998823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:42.999308 (+ 485us) server_negotiation.cc:407] Connection header received
0923 20:18:42.999479 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:42.999485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:42.999563 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:42.999673 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:43.001617 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.002367 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.004405 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.004866 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.004965 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.008347 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:43.008369 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:43.008376 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:43.008423 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:43.011534 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:43.012442 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:43.012450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:43.012453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:43.012646 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:43.013212 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:43.013217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:43.013220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:43.013602 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:43.013816 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:43.014410 (+ 594us) server_negotiation.cc:299] Negotiation successful
0923 20:18:43.014726 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:18:43.020753 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:42.999542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:44463 (local address 127.5.83.126:41617)
0923 20:18:43.001568 (+ 2026us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:43.001574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:43.001592 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:18:43.003102 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:43.003106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:43.003176 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:43.003263 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:43.005180 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.005926 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.008312 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.009117 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.009243 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.013579 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:43.013585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:43.013589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:43.013624 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:43.017378 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:43.018315 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:43.018320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:43.018324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:43.018465 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:43.018928 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:43.018933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:43.018936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:43.019213 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:43.019376 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:43.020082 (+ 706us) server_negotiation.cc:299] Negotiation successful
0923 20:18:43.020300 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1911,"thread_start_us":1804,"threads_started":1}
I20260923 20:18:43.021818 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:42.999727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:44463)
0923 20:18:43.000711 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.000761 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:43.002818 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:43.003548 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:43.003559 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:43.004269 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:43.005001 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:43.005024 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.006077 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.006086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:43.008181 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:43.008198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.009411 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.009423 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:43.009702 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.010417 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:43.010464 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:43.013315 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:43.017549 (+ 4234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:43.017558 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:43.017575 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:43.018181 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:43.018583 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:43.018588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:43.018592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:43.018796 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:43.019345 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:43.019353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:43.019715 (+ 362us) client_negotiation.cc:769] Sending connection context
0923 20:18:43.019989 (+ 274us) client_negotiation.cc:240] Negotiation successful
0923 20:18:43.020539 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":361,"threads_started":1}
I20260923 20:18:43.023423 20332 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41617
I20260923 20:18:43.023764 20332 heartbeater.cc:464] Registering TS with master...
I20260923 20:18:43.024501 20332 heartbeater.cc:511] Master 127.5.83.126:41617 requested a full tablet report, sending...
I20260923 20:18:43.026082 19884 ts_manager.cc:194] Registered new tserver with Master: dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505)
I20260923 20:18:43.027014 19884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:44463
I20260923 20:18:43.030645 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:18:43.037760 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.031553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.5.83.126:41617)
0923 20:18:43.031753 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:43.031762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:43.032182 (+ 420us) server_negotiation.cc:407] Connection header received
0923 20:18:43.032389 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:43.032393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:43.032522 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:43.032689 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:18:43.033154 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.033919 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.035538 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.035979 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.036132 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.036752 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:18:43.037190 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:18:43.037359 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:43.037420 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 20:18:43.037500 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:18:43.038754 19884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:18:43.046486 19884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:18:43.071231 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.059844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:45505 (local address 127.0.0.1:56834)
0923 20:18:43.061591 (+ 1747us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.061612 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:43.061730 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:43.062790 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:43.062797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:43.062828 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:43.063220 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:43.063233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.065219 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.065224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:43.067792 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:43.067806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.069902 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.069911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:43.070317 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.070384 (+ 67us) client_negotiation.cc:769] Sending connection context
0923 20:18:43.070547 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 20:18:43.070727 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1599,"mutex_wait_us":1,"thread_start_us":752,"threads_started":1}
I20260923 20:18:43.071834 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.059178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34380)
0923 20:18:43.060903 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.060926 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:43.061056 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:43.064479 (+ 3423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:43.064484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:43.064513 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:43.064986 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:43.065004 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.066917 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.066924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:43.068997 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:43.069012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.070969 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.070975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:43.071163 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.071202 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 20:18:43.071293 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 20:18:43.071432 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1572,"mutex_wait_us":25,"thread_start_us":2303,"threads_started":1}
I20260923 20:18:43.071950 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.059809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.5.83.65:43983)
0923 20:18:43.063959 (+ 4150us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:43.063968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:43.063993 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:18:43.064082 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:43.064089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:43.064278 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:43.064462 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:43.065247 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.066775 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.069189 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.070720 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.070882 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.071121 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:43.071401 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 20:18:43.071569 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3982,"thread_start_us":3867,"threads_started":1}
I20260923 20:18:43.072435 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.058822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:38437 (local address 127.0.0.1:52996)
0923 20:18:43.060031 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.060078 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:43.060230 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:43.064969 (+ 4739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:43.064975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:43.065005 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:43.065466 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:43.065480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.067690 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.067711 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:43.070061 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:43.070073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.071599 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.071605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:43.071778 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.071824 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 20:18:43.071907 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 20:18:43.072032 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":66,"thread_start_us":776,"threads_started":1}
I20260923 20:18:43.073007 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.060488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.5.83.67:45505)
0923 20:18:43.062300 (+ 1812us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:43.062306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:43.062338 (+ 32us) server_negotiation.cc:407] Connection header received
0923 20:18:43.062415 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:43.062420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:43.062590 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:43.062784 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:43.063446 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.065060 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.068017 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.069648 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.072073 (+ 2425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.072424 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:43.072505 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 20:18:43.072678 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1657,"thread_start_us":1495,"threads_started":1}
I20260923 20:18:43.073468 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.059075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.5.83.66:38437)
0923 20:18:43.064344 (+ 5269us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:43.064350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:43.064374 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:18:43.064491 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:43.064499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:43.064734 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:43.064968 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:43.065705 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.067544 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.070230 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.071370 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.072802 (+ 1432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.072941 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:43.072989 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 20:18:43.073146 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5106,"thread_start_us":4993,"threads_started":1}
I20260923 20:18:43.076292 20136 tablet_service.cc:1467] Processing CreateTablet for tablet 05e9c2e5c49c4da19598abe91adc16b2 (DEFAULT_TABLE table=test-table [id=f9166e57e62f47b88934e4ff31098d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:18:43.076324 20267 tablet_service.cc:1467] Processing CreateTablet for tablet 05e9c2e5c49c4da19598abe91adc16b2 (DEFAULT_TABLE table=test-table [id=f9166e57e62f47b88934e4ff31098d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:18:43.076323 20004 tablet_service.cc:1467] Processing CreateTablet for tablet 05e9c2e5c49c4da19598abe91adc16b2 (DEFAULT_TABLE table=test-table [id=f9166e57e62f47b88934e4ff31098d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:18:43.077592 20136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e9c2e5c49c4da19598abe91adc16b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:43.077731 20267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e9c2e5c49c4da19598abe91adc16b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:43.077754 20004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e9c2e5c49c4da19598abe91adc16b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:18:43.090883 20069 heartbeater.cc:503] Master 127.5.83.126:41617 was elected leader, sending a full tablet report...
I20260923 20:18:43.094195 20349 tablet_bootstrap.cc:492] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: Bootstrap starting.
I20260923 20:18:43.095055 20350 tablet_bootstrap.cc:492] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379: Bootstrap starting.
I20260923 20:18:43.096391 20348 tablet_bootstrap.cc:492] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2: Bootstrap starting.
I20260923 20:18:43.099567 20350 tablet_bootstrap.cc:654] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:43.099572 20348 tablet_bootstrap.cc:654] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:43.100649 20350 log.cc:824] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:43.100912 20348 log.cc:824] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:43.102430 20349 tablet_bootstrap.cc:654] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: Neither blocks nor log segments found. Creating new log.
I20260923 20:18:43.103820 20349 log.cc:824] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:43.103899 20350 tablet_bootstrap.cc:492] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379: No bootstrap required, opened a new log
I20260923 20:18:43.104166 20350 ts_tablet_manager.cc:1397] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:18:43.104673 20348 tablet_bootstrap.cc:492] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2: No bootstrap required, opened a new log
I20260923 20:18:43.105050 20348 ts_tablet_manager.cc:1397] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 20:18:43.107422 20349 tablet_bootstrap.cc:492] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: No bootstrap required, opened a new log
I20260923 20:18:43.107693 20349 ts_tablet_manager.cc:1397] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20260923 20:18:43.111573 20350 raft_consensus.cc:348] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:43.111928 20350 raft_consensus.cc:374] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:43.112027 20350 raft_consensus.cc:729] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd26eabc60604ad3910de89dd711c379, State: Initialized, Role: FOLLOWER
I20260923 20:18:43.112707 20350 consensus_queue.cc:260] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:43.115322 20332 heartbeater.cc:503] Master 127.5.83.126:41617 was elected leader, sending a full tablet report...
I20260923 20:18:43.116101 20350 ts_tablet_manager.cc:1428] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260923 20:18:43.116209 20348 raft_consensus.cc:348] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:43.116767 20348 raft_consensus.cc:374] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:43.116891 20348 raft_consensus.cc:729] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b0e97b6b744100b2c791fa691a56f2, State: Initialized, Role: FOLLOWER
I20260923 20:18:43.117534 20348 consensus_queue.cc:260] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:43.118702 20201 heartbeater.cc:503] Master 127.5.83.126:41617 was elected leader, sending a full tablet report...
I20260923 20:18:43.119390 20348 ts_tablet_manager.cc:1428] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260923 20:18:43.119585 20349 raft_consensus.cc:348] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:43.120007 20349 raft_consensus.cc:374] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:18:43.120162 20349 raft_consensus.cc:729] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f06b98f9bc2418e8decbbd3b059ee30, State: Initialized, Role: FOLLOWER
I20260923 20:18:43.120754 20349 consensus_queue.cc:260] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:43.122467 20349 ts_tablet_manager.cc:1428] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
W20260923 20:18:43.248147 20333 tablet.cc:2369] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:18:43.278255 20355 raft_consensus.cc:482] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:18:43.278599 20355 raft_consensus.cc:504] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
W20260923 20:18:43.279687 20202 tablet.cc:2369] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:18:43.280797 20355 leader_election.cc:288] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f06b98f9bc2418e8decbbd3b059ee30 (127.5.83.65:43983), dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505)
I20260923 20:18:43.287858 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.281529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:59857 (local address 127.5.83.67:45505)
0923 20:18:43.281721 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:43.281727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:43.282359 (+ 632us) server_negotiation.cc:407] Connection header received
0923 20:18:43.282445 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:43.282449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:43.282524 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:43.282641 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:43.283309 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.284077 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.286002 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.286821 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.286976 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.287102 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:43.287469 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 20:18:43.287596 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:18:43.287861 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.281127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:37769 (local address 127.5.83.65:43983)
0923 20:18:43.281370 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:43.281376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:43.282146 (+ 770us) server_negotiation.cc:407] Connection header received
0923 20:18:43.282332 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:43.282338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:43.282432 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:43.282585 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:18:43.283247 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.284120 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.286045 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.286816 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.286997 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.287138 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:43.287494 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 20:18:43.287615 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 20:18:43.288033 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.281228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:45505 (local address 127.5.83.66:59857)
0923 20:18:43.282159 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.282178 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:43.282339 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:43.282673 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:43.282677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:43.282704 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:43.283130 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:43.283140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.284211 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.284219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:43.285844 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:43.285866 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.287139 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.287145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:43.287316 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.287339 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:18:43.287440 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 20:18:43.287563 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":43,"thread_start_us":639,"threads_started":1}
I20260923 20:18:43.288090 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.280996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.5.83.66:37769)
0923 20:18:43.281994 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.282023 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:43.282184 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:43.282611 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:43.282617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:43.282643 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:18:43.283080 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:43.283098 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.284276 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.284280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:43.285869 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:43.285881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.287103 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.287111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:43.287321 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.287347 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 20:18:43.287451 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:18:43.287575 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":101,"thread_start_us":557,"threads_started":1}
I20260923 20:18:43.289198 20287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e9c2e5c49c4da19598abe91adc16b2" candidate_uuid: "b3b0e97b6b744100b2c791fa691a56f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd26eabc60604ad3910de89dd711c379" is_pre_election: true
I20260923 20:18:43.289191 20024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e9c2e5c49c4da19598abe91adc16b2" candidate_uuid: "b3b0e97b6b744100b2c791fa691a56f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" is_pre_election: true
I20260923 20:18:43.289687 20287 raft_consensus.cc:2454] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b0e97b6b744100b2c791fa691a56f2 in term 0.
I20260923 20:18:43.289731 20024 raft_consensus.cc:2454] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b0e97b6b744100b2c791fa691a56f2 in term 0.
I20260923 20:18:43.290439 20090 leader_election.cc:302] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6f06b98f9bc2418e8decbbd3b059ee30, b3b0e97b6b744100b2c791fa691a56f2, dd26eabc60604ad3910de89dd711c379; no voters:
I20260923 20:18:43.290853 20355 raft_consensus.cc:2781] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:18:43.290998 20355 raft_consensus.cc:482] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:18:43.291115 20355 raft_consensus.cc:3037] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:43.293394 20355 raft_consensus.cc:504] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:43.294013 20355 leader_election.cc:288] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [CANDIDATE]: Term 1 election: Requested vote from peers 6f06b98f9bc2418e8decbbd3b059ee30 (127.5.83.65:43983), dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505)
I20260923 20:18:43.294523 20287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e9c2e5c49c4da19598abe91adc16b2" candidate_uuid: "b3b0e97b6b744100b2c791fa691a56f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd26eabc60604ad3910de89dd711c379"
I20260923 20:18:43.294530 20024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e9c2e5c49c4da19598abe91adc16b2" candidate_uuid: "b3b0e97b6b744100b2c791fa691a56f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f06b98f9bc2418e8decbbd3b059ee30"
I20260923 20:18:43.294730 20287 raft_consensus.cc:3037] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:43.294745 20024 raft_consensus.cc:3037] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:18:43.297046 20287 raft_consensus.cc:2454] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b0e97b6b744100b2c791fa691a56f2 in term 1.
I20260923 20:18:43.297070 20024 raft_consensus.cc:2454] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b0e97b6b744100b2c791fa691a56f2 in term 1.
I20260923 20:18:43.297494 20090 leader_election.cc:302] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b0e97b6b744100b2c791fa691a56f2, dd26eabc60604ad3910de89dd711c379; no voters:
I20260923 20:18:43.297791 20355 raft_consensus.cc:2781] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:18:43.298240 20355 raft_consensus.cc:686] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 1 LEADER]: Becoming Leader. State: Replica: b3b0e97b6b744100b2c791fa691a56f2, State: Running, Role: LEADER
I20260923 20:18:43.298795 20355 consensus_queue.cc:237] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:43.303382 19883 catalog_manager.cc:5495] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66). New cstate: current_term: 1 leader_uuid: "b3b0e97b6b744100b2c791fa691a56f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } health_report { overall_health: UNKNOWN } } }
W20260923 20:18:43.304159 20070 tablet.cc:2369] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:18:43.371786 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.366171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53004 (local address 127.5.83.66:38437)
0923 20:18:43.366409 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:43.366413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:43.366432 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:18:43.366582 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:43.366586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:43.366667 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:43.366792 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:18:43.367100 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.367983 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.369742 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.370145 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.370240 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.370751 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:18:43.371268 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:18:43.371376 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:43.371499 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 20:18:43.371563 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 20:18:43.380121 20287 raft_consensus.cc:1261] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379 [term 1 FOLLOWER]: Refusing update from remote peer b3b0e97b6b744100b2c791fa691a56f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:18:43.380119 20024 raft_consensus.cc:1261] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Refusing update from remote peer b3b0e97b6b744100b2c791fa691a56f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:18:43.381188 20359 consensus_queue.cc:1035] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:18:43.381541 20355 consensus_queue.cc:1035] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:18:43.394616 20364 mvcc.cc:205] Tried to move back new op lower bound from 7332637586955190272 to 7332637586635558912. Current Snapshot: MvccSnapshot[applied={T|T < 7332637586955190272}]
I20260923 20:18:43.395848 20363 mvcc.cc:205] Tried to move back new op lower bound from 7332637586955190272 to 7332637586635558912. Current Snapshot: MvccSnapshot[applied={T|T < 7332637586955190272}]
I20260923 20:18:43.408285 19884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:18:43.408622 19884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:18:43.412282 19884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Sending DeleteTablet for 3 replicas of tablet 05e9c2e5c49c4da19598abe91adc16b2
I20260923 20:18:43.413698 20136 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e9c2e5c49c4da19598abe91adc16b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:18:43 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:52996
I20260923 20:18:43.413728 20004 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e9c2e5c49c4da19598abe91adc16b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:18:43 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:34380
I20260923 20:18:43.413774 20267 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e9c2e5c49c4da19598abe91adc16b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:18:43 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:56834
I20260923 20:18:43.414801 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 19921
I20260923 20:18:43.442059 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.416208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.5.83.65:43983)
0923 20:18:43.416427 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:43.416436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:43.416461 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:18:43.416566 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:43.416574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:43.416704 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:43.416832 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:43.418528 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.420052 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.421925 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.422807 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.422987 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.430953 (+ 7966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:43.430976 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:43.431005 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:43.431067 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:43.437718 (+ 6651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:43.438659 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:43.438672 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:43.438678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:43.438876 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:43.439788 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:43.439797 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:43.439803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:43.440430 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:43.440928 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:43.441173 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 20:18:43.441490 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:18:43.443163 20372 tablet_replica.cc:330] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: stopping tablet replica
I20260923 20:18:43.443218 20064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:43.452591 20371 tablet_replica.cc:330] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379: stopping tablet replica
I20260923 20:18:43.453101 20373 tablet_replica.cc:330] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2: stopping tablet replica
I20260923 20:18:43.453290 20371 raft_consensus.cc:2229] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:18:43.453517 20373 raft_consensus.cc:2229] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:18:43.454043 20371 raft_consensus.cc:2258] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:18:43.454372 20373 raft_consensus.cc:2258] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:18:43.456849 20373 ts_tablet_manager.cc:1905] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:18:43.456843 20371 ts_tablet_manager.cc:1905] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:18:43.464007 20371 ts_tablet_manager.cc:1918] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:18:43.464149 20373 ts_tablet_manager.cc:1918] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:18:43.464308 20371 log.cc:1196] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/05e9c2e5c49c4da19598abe91adc16b2
I20260923 20:18:43.464336 20373 log.cc:1196] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/05e9c2e5c49c4da19598abe91adc16b2
I20260923 20:18:43.464978 20373 ts_tablet_manager.cc:1939] T 05e9c2e5c49c4da19598abe91adc16b2 P b3b0e97b6b744100b2c791fa691a56f2: Deleting consensus metadata
I20260923 20:18:43.464975 20371 ts_tablet_manager.cc:1939] T 05e9c2e5c49c4da19598abe91adc16b2 P dd26eabc60604ad3910de89dd711c379: Deleting consensus metadata
I20260923 20:18:43.467038 19871 catalog_manager.cc:4841] TS b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437): tablet 05e9c2e5c49c4da19598abe91adc16b2 (table test-table [id=f9166e57e62f47b88934e4ff31098d1c]) successfully deleted
I20260923 20:18:43.467926 19869 catalog_manager.cc:4841] TS dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505): tablet 05e9c2e5c49c4da19598abe91adc16b2 (table test-table [id=f9166e57e62f47b88934e4ff31098d1c]) successfully deleted
I20260923 20:18:43.449617 20372 raft_consensus.cc:2229] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:18:43.573776 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 19921
I20260923 20:18:43.573776 20372 raft_consensus.cc:2258] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:18:43.592281 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 20073
W20260923 20:18:43.592195 19868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.65:43983 (error 108)
I20260923 20:18:43.594709 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.593757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34408)
0923 20:18:43.594455 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.594547 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":467,"threads_started":1}
W20260923 20:18:43.594996 19868 catalog_manager.cc:4584] TS 6f06b98f9bc2418e8decbbd3b059ee30 (127.5.83.65:43983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05e9c2e5c49c4da19598abe91adc16b2: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
I20260923 20:18:43.609058 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.593407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53018 (local address 127.5.83.66:38437)
0923 20:18:43.593618 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:43.593625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:43.593645 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:18:43.593715 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:43.593724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:43.593804 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:43.593937 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:43.595762 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.596703 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.598762 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.599216 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.599376 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.602787 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:43.602802 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:43.602816 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:43.602860 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:43.606120 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:43.606970 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:43.606977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:43.606979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:43.607117 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:43.607575 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:43.607578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:43.607580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:43.607938 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:43.608184 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:43.608517 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 20:18:43.608738 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:18:43.609746 20196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:43.648615 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.648034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34410)
0923 20:18:43.648280 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.648407 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 20:18:43.649760 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.649333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34416)
0923 20:18:43.649503 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.649592 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 20:18:43.710521 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.709956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34422)
0923 20:18:43.710225 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.710335 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260923 20:18:43.711624 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.711184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34426)
0923 20:18:43.711366 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.711453 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 20:18:43.724390 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 20073
I20260923 20:18:43.744465 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 20204
I20260923 20:18:43.758450 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.745200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.5.83.67:45505)
0923 20:18:43.745374 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:43.745380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:43.745478 (+ 98us) server_negotiation.cc:407] Connection header received
0923 20:18:43.745688 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:43.745693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:43.745764 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:43.745898 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:43.747229 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.747952 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.749367 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:43.749771 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:43.749944 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:43.752623 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:43.752637 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:43.752652 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:43.752691 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:43.755418 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:43.756175 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:43.756191 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:43.756194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:43.756340 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:43.756818 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:43.756825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:43.756829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:43.757223 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:43.757533 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:43.757869 (+ 336us) server_negotiation.cc:299] Negotiation successful
0923 20:18:43.758091 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:18:43.759073 20327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:43.792493 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.791915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34434)
0923 20:18:43.792174 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.792285 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 20:18:43.793766 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.793288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34442)
0923 20:18:43.793513 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.793597 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260923 20:18:43.851089 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 20204
I20260923 20:18:43.868224 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 19851
I20260923 20:18:43.869436 19914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:18:43.985064 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:43.984412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43983 (local address 127.0.0.1:34456)
0923 20:18:43.984657 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:43.984829 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 20:18:43.985244 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 19851
I20260923 20:18:44.008523 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:41617
--webserver_interface=127.5.83.126
--webserver_port=37499
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:41617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:18:44.352551 20380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:44.352913 20380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:44.353073 20380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:44.360453 20380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:18:44.360637 20380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:44.360751 20380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:44.360855 20380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:18:44.360934 20380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:18:44.371006 20380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:41617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:41617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=37499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:44.372398 20380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:44.373724 20380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:44.385859 20385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:44.385932 20386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:44.385886 20388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:44.387092 20380 server_base.cc:1034] running on GCE node
I20260923 20:18:44.387962 20380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:44.389604 20380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:44.390918 20380 hybrid_clock.cc:648] HybridClock initialized: now 1790194724390872 us; error 49 us; skew 500 ppm
I20260923 20:18:44.391301 20380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:44 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194724, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:44.394537 20380 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:18:44.396108 20380 webserver.cc:460] Webserver started at http://127.5.83.126:37499/ using document root <none> and password file <none>
I20260923 20:18:44.396901 20380 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:44.397058 20380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:44.402647 20380 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260923 20:18:44.405480 20395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:44.406769 20380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:18:44.406951 20380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "f975b77b24884acf9e62723bc0508af5"
format_stamp: "Formatted at 2026-09-23 20:18:38 on dist-test-slave-vl24"
server_key: "ededf893e6becdac5127c737dea63ee1"
server_key_iv: "653d62c2edd4a42d090249e8fd230f24"
server_key_version: "encryptionkey@0"
I20260923 20:18:44.407935 20380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:44.429515 20380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:44.433684 20380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:44.434007 20380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:44.453624 20380 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:41617
I20260923 20:18:44.453644 20446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:41617 every 8 connection(s)
I20260923 20:18:44.455202 20380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:18:44.462256 20447 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:18:44.464116 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 20380
I20260923 20:18:44.465023 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:43983
--local_ip_for_outbound_sockets=127.5.83.65
--tserver_master_addrs=127.5.83.126:41617
--webserver_port=37605
--webserver_interface=127.5.83.65
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:18:44.465875 20447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Bootstrap starting.
I20260923 20:18:44.492444 20447 log.cc:824] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:44.513648 20447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:18:44.514333 20447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5: Bootstrap complete.
I20260923 20:18:44.527318 20447 raft_consensus.cc:348] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } }
I20260923 20:18:44.528262 20447 raft_consensus.cc:729] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f975b77b24884acf9e62723bc0508af5, State: Initialized, Role: FOLLOWER
I20260923 20:18:44.528970 20447 consensus_queue.cc:260] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } }
I20260923 20:18:44.529215 20447 raft_consensus.cc:388] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:18:44.529343 20447 raft_consensus.cc:482] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:18:44.529541 20447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 2 FOLLOWER]: Advancing to term 3
I20260923 20:18:44.533514 20447 raft_consensus.cc:504] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } }
I20260923 20:18:44.534008 20447 leader_election.cc:302] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f975b77b24884acf9e62723bc0508af5; no voters:
I20260923 20:18:44.534735 20447 leader_election.cc:288] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 20:18:44.534857 20451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 20:18:44.535988 20451 raft_consensus.cc:686] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [term 3 LEADER]: Becoming Leader. State: Replica: f975b77b24884acf9e62723bc0508af5, State: Running, Role: LEADER
I20260923 20:18:44.536671 20451 consensus_queue.cc:237] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } }
I20260923 20:18:44.537075 20447 sys_catalog.cc:564] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:18:44.539297 20452 sys_catalog.cc:455] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f975b77b24884acf9e62723bc0508af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } } }
I20260923 20:18:44.539723 20452 sys_catalog.cc:458] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:44.543560 20453 sys_catalog.cc:455] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f975b77b24884acf9e62723bc0508af5. Latest consensus state: current_term: 3 leader_uuid: "f975b77b24884acf9e62723bc0508af5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f975b77b24884acf9e62723bc0508af5" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41617 } } }
I20260923 20:18:44.544018 20453 sys_catalog.cc:458] T 00000000000000000000000000000000 P f975b77b24884acf9e62723bc0508af5 [sys.catalog]: This master's current role is: LEADER
I20260923 20:18:44.545857 20457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:18:44.555619 20457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:18:44.557269 20457 catalog_manager.cc:1259] Loaded cluster ID: c2e37df7b9ac43e9b8206ca59f4fff9a
I20260923 20:18:44.557431 20457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 20:18:44.560105 20457 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f069978b9ad google::LogMessage::Fail() at ??:0
@ 0x7f069978e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f069978b4de google::LogMessage::Flush() at ??:0
@ 0x7f069978ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f06af1be512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f06af1bb976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f06af243610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f06af243439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f069c3d7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f069c3da91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f069c3da799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f069c3b5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f069f68f6db start_thread at ??:0
@ 0x7f069862561f clone at ??:0
W20260923 20:18:44.869197 20449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:44.869531 20449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:44.869717 20449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:44.877238 20449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:44.877414 20449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:44.877597 20449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:18:44.888077 20449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:43983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=37605
--tserver_master_addrs=127.5.83.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:44.889470 20449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:44.890820 20449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:44.904106 20488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:44.904048 20491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:44.904145 20489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:44.905098 20449 server_base.cc:1034] running on GCE node
I20260923 20:18:44.905727 20449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:44.906723 20449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:44.907938 20449 hybrid_clock.cc:648] HybridClock initialized: now 1790194724907891 us; error 43 us; skew 500 ppm
I20260923 20:18:44.908254 20449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:44 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194724, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:44.911535 20449 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:18:44.913101 20449 webserver.cc:460] Webserver started at http://127.5.83.65:37605/ using document root <none> and password file <none>
I20260923 20:18:44.913798 20449 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:44.913950 20449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:44.919507 20449 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 20:18:44.922659 20498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:44.924072 20449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:18:44.924232 20449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "6f06b98f9bc2418e8decbbd3b059ee30"
format_stamp: "Formatted at 2026-09-23 20:18:38 on dist-test-slave-vl24"
server_key: "5cb64f1427ae89acb5b223ee5ed7164e"
server_key_iv: "32f65bfa77f1e73a9c8f1a05d92fbfc8"
server_key_version: "encryptionkey@0"
I20260923 20:18:44.925308 20449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:44.955718 20449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:44.960230 20449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:44.960546 20449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:44.961567 20449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:44.963285 20505 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260923 20:18:44.968873 20449 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 20:18:44.968986 20449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260923 20:18:44.969080 20449 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 20:18:44.972070 20449 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 20:18:44.972146 20449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260923 20:18:44.972512 20505 tablet_bootstrap.cc:492] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: Bootstrap starting.
I20260923 20:18:45.000978 20505 log.cc:824] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: Log is configured to *not* fsync() on all Append() calls
I20260923 20:18:45.013571 20505 tablet_bootstrap.cc:492] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:18:45.014182 20505 tablet_bootstrap.cc:492] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: Bootstrap complete.
I20260923 20:18:45.015573 20505 ts_tablet_manager.cc:1397] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: Time spent bootstrapping tablet: real 0.043s user 0.028s sys 0.011s
I20260923 20:18:45.019570 20449 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:43983
I20260923 20:18:45.019629 20612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:43983 every 8 connection(s)
I20260923 20:18:45.021044 20449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:18:45.025688 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 20449
I20260923 20:18:45.024895 20505 raft_consensus.cc:348] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:45.026304 20505 raft_consensus.cc:729] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f06b98f9bc2418e8decbbd3b059ee30, State: Initialized, Role: FOLLOWER
I20260923 20:18:45.026391 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:38437
--local_ip_for_outbound_sockets=127.5.83.66
--tserver_master_addrs=127.5.83.126:41617
--webserver_port=43179
--webserver_interface=127.5.83.66
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:18:45.026916 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:45.024353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:36185)
0923 20:18:45.025344 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:45.025677 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":452,"threads_started":1}
I20260923 20:18:45.027107 20505 consensus_queue.cc:260] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
W20260923 20:18:45.027740 20613 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:41617 (0 consecutive failures): Network error: Failed to ping master at 127.5.83.126:41617: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
I20260923 20:18:45.029032 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:45.028429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:58405)
0923 20:18:45.028701 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:45.028836 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260923 20:18:45.030352 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:45.029759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:47533)
0923 20:18:45.030013 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:45.030139 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260923 20:18:45.030655 20613 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:18:45.033931 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:45.033522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:33703)
0923 20:18:45.033718 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:45.033790 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260923 20:18:45.034854 20505 ts_tablet_manager.cc:1428] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30: Time spent starting tablet: real 0.019s user 0.010s sys 0.010s
I20260923 20:18:45.035751 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:45.035443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:51363)
0923 20:18:45.035575 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:45.035648 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 20:18:45.378480 20617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:45.378862 20617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:45.379029 20617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:45.386618 20617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:45.386777 20617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:45.386970 20617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:18:45.397284 20617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:38437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=43179
--tserver_master_addrs=127.5.83.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:45.398716 20617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:45.400094 20617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:45.414230 20622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:45.414127 20625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:45.414173 20623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:45.415658 20617 server_base.cc:1034] running on GCE node
I20260923 20:18:45.416452 20617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:45.417579 20617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:45.418855 20617 hybrid_clock.cc:648] HybridClock initialized: now 1790194725418790 us; error 64 us; skew 500 ppm
I20260923 20:18:45.419232 20617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:45 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194725, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:45.422912 20617 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:18:45.424606 20617 webserver.cc:460] Webserver started at http://127.5.83.66:43179/ using document root <none> and password file <none>
I20260923 20:18:45.425406 20617 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:45.425552 20617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:45.432368 20617 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 20:18:45.435496 20632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:45.437093 20617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:18:45.437289 20617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "b3b0e97b6b744100b2c791fa691a56f2"
format_stamp: "Formatted at 2026-09-23 20:18:39 on dist-test-slave-vl24"
server_key: "42c6bd75fba72b8ce038611a85ac05e3"
server_key_iv: "9cae2fc5e8de0a3f7b802521f82ef63c"
server_key_version: "encryptionkey@0"
I20260923 20:18:45.438352 20617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:45.458045 20617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:45.462551 20617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:45.462852 20617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:45.463964 20617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:45.465415 20617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:45.465505 20617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:45.465672 20617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:45.465755 20617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:45.502574 20617 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:38437
I20260923 20:18:45.502610 20744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:38437 every 8 connection(s)
I20260923 20:18:45.504004 20617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:18:45.508658 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:45.506868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:57409)
0923 20:18:45.507554 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:45.507753 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":266,"threads_started":1}
W20260923 20:18:45.509305 20745 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:41617 (0 consecutive failures): Network error: Failed to ping master at 127.5.83.126:41617: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
I20260923 20:18:45.510118 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:45.509802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:48245)
0923 20:18:45.509952 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:45.510017 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 20:18:45.510944 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:45.510658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:55995)
0923 20:18:45.510771 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:45.510841 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 20:18:45.511185 20745 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:18:45.513346 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 20617
I20260923 20:18:45.513988 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:45505
--local_ip_for_outbound_sockets=127.5.83.67
--tserver_master_addrs=127.5.83.126:41617
--webserver_port=38497
--webserver_interface=127.5.83.67
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:18:45.845602 20748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:18:45.845877 20748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:18:45.846033 20748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:18:45.853242 20748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:18:45.853358 20748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:18:45.853552 20748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:18:45.863389 20748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:45505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=38497
--tserver_master_addrs=127.5.83.126:41617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:18:45.864786 20748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:18:45.866323 20748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:18:45.880268 20756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:45.880338 20754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:18:45.880306 20753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:18:45.881043 20748 server_base.cc:1034] running on GCE node
I20260923 20:18:45.881717 20748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:18:45.882674 20748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:18:45.883937 20748 hybrid_clock.cc:648] HybridClock initialized: now 1790194725883894 us; error 50 us; skew 500 ppm
I20260923 20:18:45.884295 20748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:18:45 dist-test-slave-vl24 krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194725, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:18:45.887609 20748 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:18:45.889339 20748 webserver.cc:460] Webserver started at http://127.5.83.67:38497/ using document root <none> and password file <none>
I20260923 20:18:45.890062 20748 fs_manager.cc:359] Metadata directory not provided
I20260923 20:18:45.890213 20748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:18:45.895915 20748 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260923 20:18:45.898880 20763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:45.900270 20748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:18:45.900465 20748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "dd26eabc60604ad3910de89dd711c379"
format_stamp: "Formatted at 2026-09-23 20:18:40 on dist-test-slave-vl24"
server_key: "8ffb28903d5c0fb5fddf416c4537b056"
server_key_iv: "e768d24382061484dc24faf4ad2c5540"
server_key_version: "encryptionkey@0"
I20260923 20:18:45.901553 20748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:18:45.917514 20748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:18:45.922088 20748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:18:45.922382 20748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:18:45.923372 20748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:18:45.924852 20748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:18:45.924958 20748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:45.925061 20748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:18:45.925128 20748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:18:45.959405 20748 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:45505
I20260923 20:18:45.959569 20875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:45505 every 8 connection(s)
I20260923 20:18:45.960963 20748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:18:45.965673 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:45.963855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:51853)
0923 20:18:45.964635 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:45.964845 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":285,"threads_started":1}
W20260923 20:18:45.966377 20876 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:41617 (0 consecutive failures): Network error: Failed to ping master at 127.5.83.126:41617: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
I20260923 20:18:45.967197 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:45.966894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:49489)
0923 20:18:45.967036 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:45.967098 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:18:45.967984 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:45.967701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:41025)
0923 20:18:45.967812 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:45.967877 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 20:18:45.968225 20876 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:18:45.969908 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 20748
I20260923 20:18:46.037506 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:46.036720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:51931)
0923 20:18:46.037193 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:46.037290 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":225,"threads_started":1}
I20260923 20:18:46.512598 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:46.511739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:45055)
0923 20:18:46.512206 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:46.512310 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":243,"threads_started":1}
I20260923 20:18:46.717464 20882 raft_consensus.cc:482] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:18:46.717725 20882 raft_consensus.cc:504] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:46.719144 20882 leader_election.cc:288] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437), dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505)
Sep 23 20:18:46 dist-test-slave-vl24 krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194724, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
Sep 23 20:18:46 dist-test-slave-vl24 krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194724, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:18:46.753456 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:46.719738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:33661 (local address 127.5.83.67:45505)
0923 20:18:46.733794 (+ 14056us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:46.733805 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:46.733868 (+ 63us) server_negotiation.cc:407] Connection header received
0923 20:18:46.735371 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:46.735380 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:46.735714 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:46.736048 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:46.738093 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:46.739229 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:46.741129 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:46.741778 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:46.741999 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:46.746078 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:46.746085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:46.746101 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:46.746138 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:46.749451 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:46.750382 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:46.750392 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:46.750402 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:46.750604 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:46.751155 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:46.751159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:46.751161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:46.751661 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:46.751865 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:46.752658 (+ 793us) server_negotiation.cc:299] Negotiation successful
0923 20:18:46.753036 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13893,"thread_start_us":13752,"threads_started":1}
I20260923 20:18:46.753472 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:46.719796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:56461 (local address 127.5.83.66:38437)
0923 20:18:46.733426 (+ 13630us) server_negotiation.cc:206] Beginning negotiation
0923 20:18:46.733441 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:18:46.733488 (+ 47us) server_negotiation.cc:407] Connection header received
0923 20:18:46.735195 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:18:46.735204 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:18:46.735586 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:18:46.735938 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:18:46.738094 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:46.739251 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:46.741122 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:18:46.741835 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:18:46.742026 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:46.746262 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:18:46.746268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:18:46.746281 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:18:46.746328 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:18:46.749776 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:46.750504 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:46.750510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:46.750515 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:46.750706 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:18:46.751121 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:18:46.751129 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:18:46.751133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:18:46.751662 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:18:46.751862 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0923 20:18:46.752669 (+ 807us) server_negotiation.cc:299] Negotiation successful
0923 20:18:46.753043 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13403,"thread_start_us":13261,"threads_started":1}
I20260923 20:18:46.753487 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:46.719517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:45505 (local address 127.5.83.65:33661)
0923 20:18:46.732423 (+ 12906us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:46.732440 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:46.732977 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:46.736087 (+ 3110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:46.736097 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:46.737456 (+ 1359us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:46.737888 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:46.737909 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:46.739434 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:46.739438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:46.740973 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:46.740987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:46.742011 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:46.742021 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:46.742297 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:46.743207 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:46.743233 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:46.745742 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:46.749611 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:46.749622 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:46.749641 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:46.750226 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:46.750726 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:46.750732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:46.750738 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:46.751016 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:46.751807 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:46.751815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:46.752136 (+ 321us) client_negotiation.cc:769] Sending connection context
0923 20:18:46.752398 (+ 262us) client_negotiation.cc:240] Negotiation successful
0923 20:18:46.752999 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12796,"mutex_wait_us":184,"spinlock_wait_cycles":77952,"thread_start_us":12308,"threads_started":1}
I20260923 20:18:46.753664 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:46.719478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:38437 (local address 127.5.83.65:56461)
0923 20:18:46.732188 (+ 12710us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:46.732255 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 20:18:46.733432 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:18:46.736089 (+ 2657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:18:46.736097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:18:46.737164 (+ 1067us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:18:46.737889 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:46.737909 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:46.739432 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:46.739437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:46.740977 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:18:46.740990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:18:46.742086 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:18:46.742094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:18:46.742300 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:18:46.743289 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:18:46.743306 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:18:46.745985 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:18:46.749908 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:46.749912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:46.749914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:46.750355 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:46.750823 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:18:46.750827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:18:46.750829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:18:46.750998 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:18:46.751806 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:18:46.751814 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:18:46.752136 (+ 322us) client_negotiation.cc:769] Sending connection context
0923 20:18:46.752398 (+ 262us) client_negotiation.cc:240] Negotiation successful
0923 20:18:46.753168 (+ 770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12534,"mutex_wait_us":362,"spinlock_wait_cycles":7296,"thread_start_us":13083,"threads_started":1}
W20260923 20:18:46.757015 20502 leader_election.cc:341] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437): Not found: Tablet not found: 05e9c2e5c49c4da19598abe91adc16b2
W20260923 20:18:46.757292 20500 leader_election.cc:341] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505): Not found: Tablet not found: 05e9c2e5c49c4da19598abe91adc16b2
I20260923 20:18:46.757411 20500 leader_election.cc:302] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f06b98f9bc2418e8decbbd3b059ee30; no voters: b3b0e97b6b744100b2c791fa691a56f2, dd26eabc60604ad3910de89dd711c379
I20260923 20:18:46.757844 20882 raft_consensus.cc:2726] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:18:46.969842 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:46.968886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:44137)
0923 20:18:46.969415 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:46.969576 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":282,"threads_started":1}
I20260923 20:18:47.038822 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:47.038392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:35575)
0923 20:18:47.038568 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:47.038669 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 20:18:47.514282 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:47.513593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:42569)
0923 20:18:47.514044 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:47.514128 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":229,"threads_started":1}
I20260923 20:18:47.971524 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:47.970784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:41553)
0923 20:18:47.971262 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:47.971346 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":258,"threads_started":1}
I20260923 20:18:48.040691 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:48.039834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:58329)
0923 20:18:48.040383 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:48.040481 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":261,"threads_started":1}
I20260923 20:18:48.515806 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:48.515132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:33783)
0923 20:18:48.515555 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:48.515636 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":219,"threads_started":1}
I20260923 20:18:48.877342 20892 raft_consensus.cc:482] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:18:48.877597 20892 raft_consensus.cc:504] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:48.878275 20892 leader_election.cc:288] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437), dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505)
W20260923 20:18:48.879307 20500 leader_election.cc:341] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505): Not found: Tablet not found: 05e9c2e5c49c4da19598abe91adc16b2
W20260923 20:18:48.879501 20502 leader_election.cc:341] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437): Not found: Tablet not found: 05e9c2e5c49c4da19598abe91adc16b2
I20260923 20:18:48.879690 20502 leader_election.cc:302] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f06b98f9bc2418e8decbbd3b059ee30; no voters: b3b0e97b6b744100b2c791fa691a56f2, dd26eabc60604ad3910de89dd711c379
I20260923 20:18:48.879951 20892 raft_consensus.cc:2726] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:18:48.973285 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:48.972612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:38543)
0923 20:18:48.973045 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:48.973129 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":222,"threads_started":1}
I20260923 20:18:49.042466 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:49.041683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:60961)
0923 20:18:49.042153 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:49.042258 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":238,"threads_started":1}
I20260923 20:18:49.517647 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:49.516885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:44757)
0923 20:18:49.517377 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:49.517461 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":257,"threads_started":1}
I20260923 20:18:49.975162 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:49.974316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:40553)
0923 20:18:49.974884 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:49.974977 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":317,"threads_started":1}
I20260923 20:18:50.044317 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:50.043584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:51041)
0923 20:18:50.044063 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:50.044149 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":267,"threads_started":1}
I20260923 20:18:50.519503 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:50.518675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:33165)
0923 20:18:50.519200 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:50.519310 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":285,"threads_started":1}
I20260923 20:18:50.977247 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:50.976296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:33241)
0923 20:18:50.977009 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:50.977084 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":483,"threads_started":1}
I20260923 20:18:51.045945 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:51.045269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:57359)
0923 20:18:51.045695 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:51.045782 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":243,"threads_started":1}
I20260923 20:18:51.341724 20901 raft_consensus.cc:482] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:18:51.341964 20901 raft_consensus.cc:504] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:51.342796 20901 leader_election.cc:288] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437), dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505)
W20260923 20:18:51.343719 20502 leader_election.cc:341] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437): Not found: Tablet not found: 05e9c2e5c49c4da19598abe91adc16b2
W20260923 20:18:51.343909 20500 leader_election.cc:341] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505): Not found: Tablet not found: 05e9c2e5c49c4da19598abe91adc16b2
I20260923 20:18:51.344080 20500 leader_election.cc:302] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f06b98f9bc2418e8decbbd3b059ee30; no voters: b3b0e97b6b744100b2c791fa691a56f2, dd26eabc60604ad3910de89dd711c379
I20260923 20:18:51.344337 20901 raft_consensus.cc:2726] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:18:51.521329 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:51.520609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:34997)
0923 20:18:51.521066 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:51.521150 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":232,"threads_started":1}
I20260923 20:18:51.979146 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:51.978170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:38399)
0923 20:18:51.978817 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:51.978932 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":255,"threads_started":1}
I20260923 20:18:52.047609 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:52.046889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:37777)
0923 20:18:52.047324 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:52.047407 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":237,"threads_started":1}
I20260923 20:18:52.523360 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:52.522422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:51895)
0923 20:18:52.523060 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:52.523160 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":319,"threads_started":1}
I20260923 20:18:52.981103 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:52.980121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:37041)
0923 20:18:52.980700 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:52.980814 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":237,"threads_started":1}
I20260923 20:18:53.049603 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:53.048736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:48421)
0923 20:18:53.049362 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:53.049441 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":328,"threads_started":1}
I20260923 20:18:53.525167 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:53.524416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:47689)
0923 20:18:53.524881 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:53.524983 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":262,"threads_started":1}
I20260923 20:18:53.982892 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:53.982144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:58397)
0923 20:18:53.982615 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:53.982703 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":277,"threads_started":1}
I20260923 20:18:54.051534 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:54.050638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:54395)
0923 20:18:54.051270 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:54.051350 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":321,"threads_started":1}
I20260923 20:18:54.526970 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:54.526197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:52807)
0923 20:18:54.526707 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:54.526807 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":241,"threads_started":1}
I20260923 20:18:54.984689 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:54.983773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:35887)
0923 20:18:54.984403 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:54.984480 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":393,"threads_started":1}
I20260923 20:18:55.053328 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:55.052643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:45689)
0923 20:18:55.053065 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:55.053146 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":210,"threads_started":1}
I20260923 20:18:55.528856 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:55.527997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:40077)
0923 20:18:55.528439 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:55.528518 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":246,"threads_started":1}
I20260923 20:18:55.986527 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:55.985697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:59357)
0923 20:18:55.986255 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:55.986335 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":333,"threads_started":1}
I20260923 20:18:56.055001 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:56.054269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:41317)
0923 20:18:56.054703 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:56.054791 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":227,"threads_started":1}
I20260923 20:18:56.351961 20917 raft_consensus.cc:482] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:18:56.352175 20917 raft_consensus.cc:504] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b0e97b6b744100b2c791fa691a56f2" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38437 } } peers { permanent_uuid: "6f06b98f9bc2418e8decbbd3b059ee30" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43983 } } peers { permanent_uuid: "dd26eabc60604ad3910de89dd711c379" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45505 } }
I20260923 20:18:56.352873 20917 leader_election.cc:288] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437), dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505)
W20260923 20:18:56.353837 20502 leader_election.cc:341] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b0e97b6b744100b2c791fa691a56f2 (127.5.83.66:38437): Not found: Tablet not found: 05e9c2e5c49c4da19598abe91adc16b2
W20260923 20:18:56.354007 20500 leader_election.cc:341] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dd26eabc60604ad3910de89dd711c379 (127.5.83.67:45505): Not found: Tablet not found: 05e9c2e5c49c4da19598abe91adc16b2
I20260923 20:18:56.354175 20500 leader_election.cc:302] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f06b98f9bc2418e8decbbd3b059ee30; no voters: b3b0e97b6b744100b2c791fa691a56f2, dd26eabc60604ad3910de89dd711c379
I20260923 20:18:56.354427 20917 raft_consensus.cc:2726] T 05e9c2e5c49c4da19598abe91adc16b2 P 6f06b98f9bc2418e8decbbd3b059ee30 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 20:18:56.530614 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:56.529813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:36635)
0923 20:18:56.530345 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:56.530436 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":282,"threads_started":1}
I20260923 20:18:56.988111 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:56.987409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:59651)
0923 20:18:56.987872 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:56.987954 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":232,"threads_started":1}
I20260923 20:18:57.056782 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:57.055933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:33539)
0923 20:18:57.056414 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:57.056496 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":275,"threads_started":1}
I20260923 20:18:57.532408 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:57.531638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:58125)
0923 20:18:57.532143 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:57.532230 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":249,"threads_started":1}
I20260923 20:18:57.989781 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:57.989086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:59931)
0923 20:18:57.989533 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:57.989613 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
I20260923 20:18:58.058549 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:58.057781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:40377)
0923 20:18:58.058286 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:58.058378 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":243,"threads_started":1}
I20260923 20:18:58.534317 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:58.533425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:53803)
0923 20:18:58.533964 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:58.534117 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":247,"threads_started":1}
I20260923 20:18:58.991624 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:58.990871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:46517)
0923 20:18:58.991351 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:58.991461 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":255,"threads_started":1}
I20260923 20:18:59.060302 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:59.059564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:33857)
0923 20:18:59.060033 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:59.060121 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":252,"threads_started":1}
I20260923 20:18:59.536098 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:59.535286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:39487)
0923 20:18:59.535802 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:59.535891 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":273,"threads_started":1}
I20260923 20:18:59.993477 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:18:59.992654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:55033)
0923 20:18:59.993182 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 20:18:59.993283 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":292,"threads_started":1}
I20260923 20:19:00.062355 20929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:00.061620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.65:33413)
0923 20:19:00.062084 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:00.062178 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":219,"threads_started":1}
I20260923 20:19:00.538179 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:00.537450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.66:38273)
0923 20:19:00.537890 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:00.538015 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":217,"threads_started":1}
I20260923 20:19:00.976408 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 20449
I20260923 20:19:00.995615 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:00.994749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41617 (local address 127.5.83.67:40843)
0923 20:19:00.995221 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:00.995404 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":244,"threads_started":1}
I20260923 20:19:00.996395 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:00.977794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.5.83.65:43983)
0923 20:19:00.978387 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:00.978399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:00.978438 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:19:00.978535 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:00.978545 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:00.978855 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:00.979064 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:00.980901 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:00.981868 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:00.983146 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:00.983692 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:00.983787 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:00.990353 (+ 6566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:00.990365 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:00.990379 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:00.990415 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:00.993323 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:00.994057 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:00.994063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:00.994065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:00.994193 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:00.994620 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:00.994625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:00.994628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:00.995156 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:00.995400 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:00.995883 (+ 483us) server_negotiation.cc:299] Negotiation successful
0923 20:19:00.996153 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":697,"threads_started":1}
I20260923 20:19:00.997722 20608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:19:01.093813 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 20449
I20260923 20:19:01.110288 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 20617
I20260923 20:19:01.123592 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:01.111116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.5.83.66:38437)
0923 20:19:01.111615 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:01.111620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:01.111636 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:19:01.111693 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:01.111697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:01.111754 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:01.111844 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:01.113397 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:01.114215 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:01.115446 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:01.115925 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:01.116024 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:01.118765 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:01.118769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:01.118773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:01.118805 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:01.121385 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:01.122004 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:01.122008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:01.122010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:01.122140 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:01.122444 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:01.122446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:01.122448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:01.122708 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:01.122906 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:01.123132 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 20:19:01.123295 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":291,"threads_started":1}
I20260923 20:19:01.124271 20740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:19:01.213477 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 20617
I20260923 20:19:01.232779 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 20748
I20260923 20:19:01.247148 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:01.233591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.5.83.67:45505)
0923 20:19:01.234139 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:01.234144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:01.234160 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:19:01.234214 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:01.234217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:01.234286 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:01.234414 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:01.235871 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:01.236891 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:01.238238 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:01.238677 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:01.238836 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:01.241681 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:01.241686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:01.241689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:01.241723 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:01.244224 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:01.245050 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:01.245058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:01.245065 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:01.245253 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:01.245628 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:01.245634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:01.245637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:01.245912 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:01.246232 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:01.246544 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 20:19:01.246764 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":253,"threads_started":1}
I20260923 20:19:01.247840 20871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:19:01.331544 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 20748
2026-09-23T20:19:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24737 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:19:01 dist-test-slave-vl24 krb5kdc[20941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:19:01 dist-test-slave-vl24 krb5kdc[20941](info): set up 2 sockets
Sep 23 20:19:01 dist-test-slave-vl24 krb5kdc[20941](info): commencing operation
krb5kdc: starting...
W20260923 20:19:02.446164 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:19:02 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:19:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:19:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:19:02.711692 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:46799
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:46799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:19:03.049412 20954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:19:03.049717 20954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:19:03.049904 20954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:19:03.057387 20954 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 20:19:03.057540 20954 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 20:19:03.057616 20954 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 20:19:03.057718 20954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:19:03.057793 20954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:19:03.057914 20954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:19:03.057991 20954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:19:03.058074 20954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:19:03.068185 20954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:46799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:46799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:19:03.069536 20954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:19:03.070906 20954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:19:03.082690 20960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:03.082760 20962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:03.082790 20959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:19:03.083439 20954 server_base.cc:1034] running on GCE node
I20260923 20:19:03.084192 20954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:19:03.085633 20954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:19:03.086870 20954 hybrid_clock.cc:648] HybridClock initialized: now 1790194743086811 us; error 27 us; skew 500 ppm
I20260923 20:19:03.087183 20954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:19:03 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194743, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:19:03.090480 20954 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:19:03.092041 20954 webserver.cc:460] Webserver started at http://127.5.83.126:42513/ using document root <none> and password file <none>
I20260923 20:19:03.092849 20954 fs_manager.cc:359] Metadata directory not provided
I20260923 20:19:03.092964 20954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:19:03.093308 20954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:19:03.096349 20954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "e277bfca93634e4fbd79f266c2193cb7"
format_stamp: "Formatted at 2026-09-23 20:19:03 on dist-test-slave-vl24"
server_key: "4b2ed2c783707f3e3092ff1f66466f7c"
server_key_iv: "182b616e179a9d36483cc0881c51da7c"
server_key_version: "encryptionkey@0"
I20260923 20:19:03.097256 20954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "e277bfca93634e4fbd79f266c2193cb7"
format_stamp: "Formatted at 2026-09-23 20:19:03 on dist-test-slave-vl24"
server_key: "4b2ed2c783707f3e3092ff1f66466f7c"
server_key_iv: "182b616e179a9d36483cc0881c51da7c"
server_key_version: "encryptionkey@0"
I20260923 20:19:03.102721 20954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:19:03.106653 20969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:03.108127 20954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:19:03.108348 20954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "e277bfca93634e4fbd79f266c2193cb7"
format_stamp: "Formatted at 2026-09-23 20:19:03 on dist-test-slave-vl24"
server_key: "4b2ed2c783707f3e3092ff1f66466f7c"
server_key_iv: "182b616e179a9d36483cc0881c51da7c"
server_key_version: "encryptionkey@0"
I20260923 20:19:03.108623 20954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:19:03.121671 20954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:19:03.125743 20954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:19:03.126026 20954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:19:03.146350 20954 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:46799
I20260923 20:19:03.146405 21020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:46799 every 8 connection(s)
I20260923 20:19:03.147899 20954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:19:03.151880 21021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:19:03.158176 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 20954
I20260923 20:19:03.158427 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:19:03.159070 5453 external_mini_cluster.cc:1428] Setting key 6104f8eda95a55141ab8d5354c6c4556
I20260923 20:19:03.164927 21021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Bootstrap starting.
I20260923 20:19:03.169536 21021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Neither blocks nor log segments found. Creating new log.
I20260923 20:19:03.171217 21021 log.cc:824] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Log is configured to *not* fsync() on all Append() calls
I20260923 20:19:03.175396 21021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: No bootstrap required, opened a new log
Sep 23 20:19:03 dist-test-slave-vl24 krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:19:03.184249 21021 raft_consensus.cc:348] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } }
I20260923 20:19:03.184722 21021 raft_consensus.cc:374] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:19:03.184842 21021 raft_consensus.cc:729] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e277bfca93634e4fbd79f266c2193cb7, State: Initialized, Role: FOLLOWER
I20260923 20:19:03.185575 21021 consensus_queue.cc:260] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } }
I20260923 20:19:03.185899 21021 raft_consensus.cc:388] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:19:03.186081 21021 raft_consensus.cc:482] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:19:03.186288 21021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:19:03.189853 21021 raft_consensus.cc:504] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } }
I20260923 20:19:03.190487 21021 leader_election.cc:302] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e277bfca93634e4fbd79f266c2193cb7; no voters:
I20260923 20:19:03.191164 21021 leader_election.cc:288] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:19:03.191289 21026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:19:03.192685 21026 raft_consensus.cc:686] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 1 LEADER]: Becoming Leader. State: Replica: e277bfca93634e4fbd79f266c2193cb7, State: Running, Role: LEADER
I20260923 20:19:03.193302 21026 consensus_queue.cc:237] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } }
I20260923 20:19:03.193764 21021 sys_catalog.cc:564] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:19:03.194003 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:03.161294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.5.83.126:46799)
0923 20:19:03.170361 (+ 9067us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:03.170379 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:03.170433 (+ 54us) server_negotiation.cc:407] Connection header received
0923 20:19:03.173254 (+ 2821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:03.173279 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:03.173799 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:03.174252 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:03.175690 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:03.177882 (+ 2192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:03.179278 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:03.180382 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:03.180613 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:03.184744 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:03.184757 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:03.184777 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:03.184837 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:03.189007 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:03.189741 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:03.189750 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:03.189761 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:03.189996 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:03.190427 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:03.190433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:03.190436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:03.191233 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:03.191669 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:03.192096 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 20:19:03.192638 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8772,"thread_start_us":8493,"threads_started":1}
I20260923 20:19:03.196372 21027 sys_catalog.cc:455] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e277bfca93634e4fbd79f266c2193cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } } }
I20260923 20:19:03.196921 21027 sys_catalog.cc:458] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: This master's current role is: LEADER
I20260923 20:19:03.198558 21028 sys_catalog.cc:455] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e277bfca93634e4fbd79f266c2193cb7. Latest consensus state: current_term: 1 leader_uuid: "e277bfca93634e4fbd79f266c2193cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } } }
I20260923 20:19:03.199003 21028 sys_catalog.cc:458] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: This master's current role is: LEADER
I20260923 20:19:03.200690 21031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:19:03.205686 21031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:19:03.215060 21031 catalog_manager.cc:1347] Generated new cluster ID: aa81963a24b94261a8e6fea57ff2f34a
I20260923 20:19:03.215262 21031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:19:03.233134 21031 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:19:03.234519 21031 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:19:03.251042 21031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Generated new TSK 0
I20260923 20:19:03.252238 21031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:19:03.335783 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46799
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:19:03.704169 21047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:19:03.704505 21047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:19:03.704710 21047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:19:03.712049 21047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:19:03.712201 21047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:19:03.712392 21047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:19:03.722312 21047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:19:03.723609 21047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:19:03.724943 21047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:19:03.738358 21052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:03.738337 21055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:03.738382 21053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:19:03.739367 21047 server_base.cc:1034] running on GCE node
I20260923 20:19:03.739950 21047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:19:03.740901 21047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:19:03.742139 21047 hybrid_clock.cc:648] HybridClock initialized: now 1790194743742093 us; error 38 us; skew 500 ppm
I20260923 20:19:03.742465 21047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:19:03 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194743, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:19:03.745594 21047 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:19:03.747097 21047 webserver.cc:460] Webserver started at http://127.5.83.65:42753/ using document root <none> and password file <none>
I20260923 20:19:03.747790 21047 fs_manager.cc:359] Metadata directory not provided
I20260923 20:19:03.747937 21047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:19:03.748250 21047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:19:03.751552 21047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "6448f7afb688403d9ac2c8f667355499"
format_stamp: "Formatted at 2026-09-23 20:19:03 on dist-test-slave-vl24"
server_key: "424d1305d2e83e904fa29f1d1c1b8522"
server_key_iv: "b2b438ffc6a3d23404b3ffea1f22fb67"
server_key_version: "encryptionkey@0"
I20260923 20:19:03.752352 21047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "6448f7afb688403d9ac2c8f667355499"
format_stamp: "Formatted at 2026-09-23 20:19:03 on dist-test-slave-vl24"
server_key: "424d1305d2e83e904fa29f1d1c1b8522"
server_key_iv: "b2b438ffc6a3d23404b3ffea1f22fb67"
server_key_version: "encryptionkey@0"
I20260923 20:19:03.757920 21047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:19:03.761483 21062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:03.762728 21047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:19:03.762902 21047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "6448f7afb688403d9ac2c8f667355499"
format_stamp: "Formatted at 2026-09-23 20:19:03 on dist-test-slave-vl24"
server_key: "424d1305d2e83e904fa29f1d1c1b8522"
server_key_iv: "b2b438ffc6a3d23404b3ffea1f22fb67"
server_key_version: "encryptionkey@0"
I20260923 20:19:03.763109 21047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:19:03.779826 21047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:19:03.783892 21047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:19:03.784196 21047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:19:03.785211 21047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:19:03.786551 21047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:19:03.786669 21047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:03.786811 21047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:19:03.786883 21047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:03.821614 21047 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:43323
I20260923 20:19:03.821671 21174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:43323 every 8 connection(s)
I20260923 20:19:03.822909 21047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:19:03.827061 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 21047
I20260923 20:19:03.827294 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:19:03.828156 5453 external_mini_cluster.cc:1428] Setting key 6867392ff8c214ba6588b5373631af08
Sep 23 20:19:03 dist-test-slave-vl24 krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194743, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:19:03.846910 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:03.825442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:38427 (local address 127.5.83.126:46799)
0923 20:19:03.825937 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:03.825944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:03.826947 (+ 1003us) server_negotiation.cc:407] Connection header received
0923 20:19:03.829421 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:03.829425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:03.829490 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:03.829574 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:03.831576 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:03.832300 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:03.834855 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:03.835251 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:03.835346 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:03.839682 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:03.839688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:03.839691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:03.839725 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:03.843305 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:03.844104 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:03.844113 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:03.844119 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:03.844311 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:03.844799 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:03.844807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:03.844813 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:03.845248 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:03.845415 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:03.846103 (+ 688us) server_negotiation.cc:299] Negotiation successful
0923 20:19:03.846373 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":224,"threads_started":1}
I20260923 20:19:03.847501 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:03.825955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:38427)
0923 20:19:03.826751 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:03.826805 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:03.829100 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:03.829841 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:03.829852 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:03.830530 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:19:03.831371 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:03.831400 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:03.832466 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:03.832482 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:03.834704 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:03.834723 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:03.835502 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:03.835512 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:03.835849 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:03.836666 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:19:03.836702 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:19:03.839431 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:19:03.843467 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:19:03.843477 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:19:03.843495 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:19:03.843939 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:19:03.844427 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:19:03.844430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:19:03.844433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:19:03.844674 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:19:03.845374 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:19:03.845380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:19:03.845724 (+ 344us) client_negotiation.cc:769] Sending connection context
0923 20:19:03.845954 (+ 230us) client_negotiation.cc:240] Negotiation successful
0923 20:19:03.846359 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":271,"threads_started":1}
I20260923 20:19:03.849191 21175 heartbeater.cc:347] Connected to a master server at 127.5.83.126:46799
I20260923 20:19:03.849509 21175 heartbeater.cc:464] Registering TS with master...
I20260923 20:19:03.850253 21175 heartbeater.cc:511] Master 127.5.83.126:46799 requested a full tablet report, sending...
I20260923 20:19:03.852442 20986 ts_manager.cc:194] Registered new tserver with Master: 6448f7afb688403d9ac2c8f667355499 (127.5.83.65:43323)
I20260923 20:19:03.853951 20986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:38427
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:19:03.888247 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46799
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:19:04.227010 21181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:19:04.227351 21181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:19:04.227519 21181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:19:04.234773 21181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:19:04.234948 21181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:19:04.235150 21181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:19:04.245055 21181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:19:04.246407 21181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:19:04.247730 21181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:19:04.261729 21187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:04.261662 21189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:04.261694 21186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:19:04.262751 21181 server_base.cc:1034] running on GCE node
I20260923 20:19:04.263530 21181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:19:04.264616 21181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:19:04.265865 21181 hybrid_clock.cc:648] HybridClock initialized: now 1790194744265809 us; error 54 us; skew 500 ppm
I20260923 20:19:04.266196 21181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:19:04 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194744, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:19:04.269560 21181 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:19:04.271044 21181 webserver.cc:460] Webserver started at http://127.5.83.66:40207/ using document root <none> and password file <none>
I20260923 20:19:04.271775 21181 fs_manager.cc:359] Metadata directory not provided
I20260923 20:19:04.271924 21181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:19:04.272262 21181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:19:04.275240 21181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "c0bfd44eba284296873dbd9465c781f3"
format_stamp: "Formatted at 2026-09-23 20:19:04 on dist-test-slave-vl24"
server_key: "67d5abec5c07b0ed8cfdd9abb97b5e44"
server_key_iv: "b20c2140cc78cd468e0f964c7bd7fdc2"
server_key_version: "encryptionkey@0"
I20260923 20:19:04.276037 21181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "c0bfd44eba284296873dbd9465c781f3"
format_stamp: "Formatted at 2026-09-23 20:19:04 on dist-test-slave-vl24"
server_key: "67d5abec5c07b0ed8cfdd9abb97b5e44"
server_key_iv: "b20c2140cc78cd468e0f964c7bd7fdc2"
server_key_version: "encryptionkey@0"
I20260923 20:19:04.281464 21181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:19:04.284981 21196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:04.286229 21181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:19:04.286409 21181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "c0bfd44eba284296873dbd9465c781f3"
format_stamp: "Formatted at 2026-09-23 20:19:04 on dist-test-slave-vl24"
server_key: "67d5abec5c07b0ed8cfdd9abb97b5e44"
server_key_iv: "b20c2140cc78cd468e0f964c7bd7fdc2"
server_key_version: "encryptionkey@0"
I20260923 20:19:04.286628 21181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:19:04.301554 21181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:19:04.305569 21181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:19:04.305869 21181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:19:04.306843 21181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:19:04.308145 21181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:19:04.308251 21181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:04.308400 21181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:19:04.308487 21181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:04.342192 21181 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:42385
I20260923 20:19:04.342237 21308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:42385 every 8 connection(s)
I20260923 20:19:04.343550 21181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:19:04.343863 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 21181
I20260923 20:19:04.344046 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:19:04.344655 5453 external_mini_cluster.cc:1428] Setting key 4dff81c6762d9ac7a6d7f3819351746e
Sep 23 20:19:04 dist-test-slave-vl24 krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194744, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:19:04.366915 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:04.346362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:44411 (local address 127.5.83.126:46799)
0923 20:19:04.346572 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:04.346578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:04.347972 (+ 1394us) server_negotiation.cc:407] Connection header received
0923 20:19:04.351277 (+ 3305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:04.351281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:04.351354 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:04.351432 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:04.353611 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.354303 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.356780 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.357201 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.357296 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:04.361065 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:04.361069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:04.361072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:04.361104 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:04.363713 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:04.364444 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:04.364454 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:04.364456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:04.364660 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:04.365125 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:04.365130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:04.365133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:04.365402 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:04.365575 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:04.366283 (+ 708us) server_negotiation.cc:299] Negotiation successful
0923 20:19:04.366522 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:19:04.367959 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:04.346907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:44411)
0923 20:19:04.347789 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:04.347852 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:04.350938 (+ 3086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:04.351707 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:04.351721 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:04.352568 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:19:04.353413 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:04.353442 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.354464 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.354472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:04.356641 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:04.356660 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.357439 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.357448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:04.357776 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:04.358525 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:19:04.358547 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:19:04.360833 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:19:04.363857 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:19:04.363866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:19:04.363882 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:19:04.364310 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:19:04.364795 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:19:04.364800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:19:04.364804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:19:04.365000 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:19:04.365541 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:19:04.365549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:19:04.365926 (+ 377us) client_negotiation.cc:769] Sending connection context
0923 20:19:04.366188 (+ 262us) client_negotiation.cc:240] Negotiation successful
0923 20:19:04.366710 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":320,"threads_started":1}
I20260923 20:19:04.369709 21309 heartbeater.cc:347] Connected to a master server at 127.5.83.126:46799
I20260923 20:19:04.370077 21309 heartbeater.cc:464] Registering TS with master...
I20260923 20:19:04.371044 21309 heartbeater.cc:511] Master 127.5.83.126:46799 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
I20260923 20:19:04.372781 20986 ts_manager.cc:194] Registered new tserver with Master: c0bfd44eba284296873dbd9465c781f3 (127.5.83.66:42385)
I20260923 20:19:04.373636 20986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:44411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:19:04.404191 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46799
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:19:04.734793 21314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:19:04.735126 21314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:19:04.735296 21314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:19:04.742758 21314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:19:04.742935 21314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:19:04.743116 21314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:19:04.753012 21314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:46799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:19:04.754397 21314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:19:04.755716 21314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:19:04.769454 21322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:04.769429 21320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:04.769526 21319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:19:04.770221 21314 server_base.cc:1034] running on GCE node
I20260923 20:19:04.770812 21314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:19:04.771715 21314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:19:04.772920 21314 hybrid_clock.cc:648] HybridClock initialized: now 1790194744772868 us; error 44 us; skew 500 ppm
I20260923 20:19:04.773195 21314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:19:04 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194744, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:19:04.776880 21314 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:19:04.778298 21314 webserver.cc:460] Webserver started at http://127.5.83.67:38517/ using document root <none> and password file <none>
I20260923 20:19:04.778993 21314 fs_manager.cc:359] Metadata directory not provided
I20260923 20:19:04.779119 21314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:19:04.779435 21314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:19:04.782332 21314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "bd49ffff1b4b41beb52ee5ec19a552e7"
format_stamp: "Formatted at 2026-09-23 20:19:04 on dist-test-slave-vl24"
server_key: "92453e8fe57b539b0859c0958314d55a"
server_key_iv: "2ca4da513c0e7e0255c7e4d382555fcb"
server_key_version: "encryptionkey@0"
I20260923 20:19:04.783071 21314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "bd49ffff1b4b41beb52ee5ec19a552e7"
format_stamp: "Formatted at 2026-09-23 20:19:04 on dist-test-slave-vl24"
server_key: "92453e8fe57b539b0859c0958314d55a"
server_key_iv: "2ca4da513c0e7e0255c7e4d382555fcb"
server_key_version: "encryptionkey@0"
I20260923 20:19:04.788303 21314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 20:19:04.791894 21329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:04.793406 21314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:19:04.793581 21314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "bd49ffff1b4b41beb52ee5ec19a552e7"
format_stamp: "Formatted at 2026-09-23 20:19:04 on dist-test-slave-vl24"
server_key: "92453e8fe57b539b0859c0958314d55a"
server_key_iv: "2ca4da513c0e7e0255c7e4d382555fcb"
server_key_version: "encryptionkey@0"
I20260923 20:19:04.793831 21314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:19:04.809599 21314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:19:04.813753 21314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:19:04.814033 21314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:19:04.814977 21314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:19:04.816283 21314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:19:04.816354 21314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:04.816489 21314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:19:04.816617 21314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260923 20:19:04.851058 21314 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:34787
I20260923 20:19:04.851109 21441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:34787 every 8 connection(s)
I20260923 20:19:04.852360 21314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:19:04.856959 21175 heartbeater.cc:503] Master 127.5.83.126:46799 was elected leader, sending a full tablet report...
I20260923 20:19:04.860105 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 21314
I20260923 20:19:04.860307 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:19:04.860888 5453 external_mini_cluster.cc:1428] Setting key b86f14a5cf5179b12273eabfa93eff70
Sep 23 20:19:04 dist-test-slave-vl24 krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194744, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:19:04.877436 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:04.855058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:45471 (local address 127.5.83.126:46799)
0923 20:19:04.855270 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:04.855276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:04.856845 (+ 1569us) server_negotiation.cc:407] Connection header received
0923 20:19:04.858868 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:04.858872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:04.858941 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:04.859029 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:04.860813 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.861744 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.863509 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.864061 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.864158 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:04.867995 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:04.868006 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:04.868013 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:04.868064 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:04.872965 (+ 4901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:04.874115 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:04.874122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:04.874126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:04.874300 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:04.874889 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:04.874895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:04.874899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:04.875334 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:04.875508 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:04.876841 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0923 20:19:04.877080 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 20:19:04.878697 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:04.855792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:45471)
0923 20:19:04.856674 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:04.856719 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:04.858621 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:04.859343 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:04.859351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:04.859902 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:19:04.860641 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:04.860663 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.861904 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.861910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:04.863374 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:04.863386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.864247 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.864252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:04.864451 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:04.865212 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:19:04.865236 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:19:04.867664 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:19:04.873175 (+ 5511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:19:04.873194 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:19:04.873218 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:19:04.873941 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:19:04.874457 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:19:04.874462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:19:04.874467 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:19:04.874738 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:19:04.875757 (+ 1019us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:19:04.875767 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:19:04.876349 (+ 582us) client_negotiation.cc:769] Sending connection context
0923 20:19:04.876745 (+ 396us) client_negotiation.cc:240] Negotiation successful
0923 20:19:04.877306 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":361,"threads_started":1}
I20260923 20:19:04.880501 21442 heartbeater.cc:347] Connected to a master server at 127.5.83.126:46799
I20260923 20:19:04.880847 21442 heartbeater.cc:464] Registering TS with master...
I20260923 20:19:04.881640 21442 heartbeater.cc:511] Master 127.5.83.126:46799 requested a full tablet report, sending...
I20260923 20:19:04.883750 20986 ts_manager.cc:194] Registered new tserver with Master: bd49ffff1b4b41beb52ee5ec19a552e7 (127.5.83.67:34787)
I20260923 20:19:04.884761 20986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:45471
I20260923 20:19:04.890223 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:19:04.914943 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:04.896200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59996 (local address 127.5.83.126:46799)
0923 20:19:04.896411 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:04.896421 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:04.897002 (+ 581us) server_negotiation.cc:407] Connection header received
0923 20:19:04.897204 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:04.897212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:04.897301 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:04.897423 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:04.899101 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.900377 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.902776 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.903561 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.903710 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:04.907438 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:04.907444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:04.907449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:04.907494 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:04.911473 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:04.912378 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:04.912387 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:04.912392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:04.912621 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:04.913033 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:04.913040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:04.913045 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:04.913583 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:04.913885 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:04.914212 (+ 327us) server_negotiation.cc:299] Negotiation successful
0923 20:19:04.914493 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 20:19:04.919265 20986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:19:04.926267 20986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:19:04.949388 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:04.938919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.5.83.66:42385)
0923 20:19:04.941013 (+ 2094us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:04.941023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:04.941052 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:19:04.941151 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:04.941161 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:04.941411 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:04.941611 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:19:04.942272 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.943888 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.946745 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.948223 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.948362 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:04.948601 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:04.948863 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 20:19:04.949038 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1927,"thread_start_us":1799,"threads_started":1}
I20260923 20:19:04.949460 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:04.938771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:42385 (local address 127.0.0.1:35608)
0923 20:19:04.939639 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:04.939673 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:04.939847 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:04.941630 (+ 1783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:04.941638 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:04.941667 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:19:04.942074 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:04.942089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.944044 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.944051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:04.946542 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:04.946561 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.948414 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.948420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:04.948640 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:04.948685 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 20:19:04.948821 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 20:19:04.949021 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":4,"thread_start_us":491,"threads_started":1}
I20260923 20:19:04.950485 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:04.938904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:34787 (local address 127.0.0.1:35078)
0923 20:19:04.940916 (+ 2012us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:04.940934 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:04.941047 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:04.943203 (+ 2156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:04.943207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:04.943229 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:19:04.943621 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:04.943634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.945575 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.945582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:04.947647 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:04.947658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.949608 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.949614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:04.949783 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:04.949815 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:19:04.949904 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 20:19:04.950035 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1897,"mutex_wait_us":72,"thread_start_us":1761,"threads_started":1}
I20260923 20:19:04.951020 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:04.941080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43323 (local address 127.0.0.1:51308)
0923 20:19:04.942937 (+ 1857us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:04.942953 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:04.943064 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:04.943825 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:04.943831 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:04.943854 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:19:04.944268 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:04.944280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.945663 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.945669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:04.947997 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:04.948012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.950195 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.950201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:04.950369 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:04.950411 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 20:19:04.950513 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 20:19:04.950655 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1712,"mutex_wait_us":40,"thread_start_us":1398,"threads_started":1}
I20260923 20:19:04.951313 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:04.941488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.5.83.65:43323)
0923 20:19:04.942311 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:04.942321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:04.943173 (+ 852us) server_negotiation.cc:407] Connection header received
0923 20:19:04.943270 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:04.943277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:04.943467 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:04.943621 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:19:04.944446 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.945518 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.948161 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.949698 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.949815 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:04.950053 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:04.950797 (+ 744us) server_negotiation.cc:299] Negotiation successful
0923 20:19:04.950976 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":481,"threads_started":1}
I20260923 20:19:04.953404 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:04.939671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.5.83.67:34787)
0923 20:19:04.942330 (+ 2659us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:04.942342 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:04.942385 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:19:04.942500 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:04.942506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:04.942738 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:04.942929 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:19:04.943826 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.945425 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.947797 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:04.949112 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:04.949244 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:04.949464 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:04.952883 (+ 3419us) server_negotiation.cc:299] Negotiation successful
0923 20:19:04.953077 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1530,"thread_start_us":1507,"threads_started":1}
I20260923 20:19:04.953363 21244 tablet_service.cc:1467] Processing CreateTablet for tablet 9120e360b57441ecb02e07e813679fe4 (DEFAULT_TABLE table=test-table [id=39a30e19b35c40f1aa1ed6d17922dfb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:19:04.954792 21244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9120e360b57441ecb02e07e813679fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:19:04.954605 21110 tablet_service.cc:1467] Processing CreateTablet for tablet 9120e360b57441ecb02e07e813679fe4 (DEFAULT_TABLE table=test-table [id=39a30e19b35c40f1aa1ed6d17922dfb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:19:04.956007 21110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9120e360b57441ecb02e07e813679fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:19:04.956939 21377 tablet_service.cc:1467] Processing CreateTablet for tablet 9120e360b57441ecb02e07e813679fe4 (DEFAULT_TABLE table=test-table [id=39a30e19b35c40f1aa1ed6d17922dfb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:19:04.958262 21377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9120e360b57441ecb02e07e813679fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:19:04.966670 21460 tablet_bootstrap.cc:492] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3: Bootstrap starting.
I20260923 20:19:04.970026 21460 tablet_bootstrap.cc:654] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3: Neither blocks nor log segments found. Creating new log.
I20260923 20:19:04.970325 21461 tablet_bootstrap.cc:492] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499: Bootstrap starting.
I20260923 20:19:04.971851 21460 log.cc:824] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3: Log is configured to *not* fsync() on all Append() calls
I20260923 20:19:04.973989 21462 tablet_bootstrap.cc:492] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7: Bootstrap starting.
I20260923 20:19:04.974231 21461 tablet_bootstrap.cc:654] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499: Neither blocks nor log segments found. Creating new log.
I20260923 20:19:04.975329 21460 tablet_bootstrap.cc:492] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3: No bootstrap required, opened a new log
I20260923 20:19:04.975493 21461 log.cc:824] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499: Log is configured to *not* fsync() on all Append() calls
I20260923 20:19:04.975626 21460 ts_tablet_manager.cc:1397] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260923 20:19:04.976997 21462 tablet_bootstrap.cc:654] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7: Neither blocks nor log segments found. Creating new log.
I20260923 20:19:04.977913 21462 log.cc:824] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7: Log is configured to *not* fsync() on all Append() calls
I20260923 20:19:04.978535 21461 tablet_bootstrap.cc:492] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499: No bootstrap required, opened a new log
I20260923 20:19:04.978797 21461 ts_tablet_manager.cc:1397] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:19:04.980684 21462 tablet_bootstrap.cc:492] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7: No bootstrap required, opened a new log
I20260923 20:19:04.980962 21462 ts_tablet_manager.cc:1397] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260923 20:19:04.986217 21460 raft_consensus.cc:348] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } }
I20260923 20:19:04.986672 21460 raft_consensus.cc:374] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:19:04.986820 21460 raft_consensus.cc:729] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bfd44eba284296873dbd9465c781f3, State: Initialized, Role: FOLLOWER
I20260923 20:19:04.987478 21460 consensus_queue.cc:260] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } }
I20260923 20:19:04.987819 21462 raft_consensus.cc:348] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } }
I20260923 20:19:04.988300 21462 raft_consensus.cc:374] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:19:04.988442 21462 raft_consensus.cc:729] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd49ffff1b4b41beb52ee5ec19a552e7, State: Initialized, Role: FOLLOWER
I20260923 20:19:04.988765 21309 heartbeater.cc:503] Master 127.5.83.126:46799 was elected leader, sending a full tablet report...
I20260923 20:19:04.989087 21462 consensus_queue.cc:260] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } }
I20260923 20:19:04.989353 21460 ts_tablet_manager.cc:1428] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260923 20:19:04.990192 21442 heartbeater.cc:503] Master 127.5.83.126:46799 was elected leader, sending a full tablet report...
I20260923 20:19:04.990274 21461 raft_consensus.cc:348] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } }
I20260923 20:19:04.990751 21462 ts_tablet_manager.cc:1428] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260923 20:19:04.990868 21461 raft_consensus.cc:374] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:19:04.991031 21461 raft_consensus.cc:729] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6448f7afb688403d9ac2c8f667355499, State: Initialized, Role: FOLLOWER
I20260923 20:19:04.991662 21461 consensus_queue.cc:260] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } }
I20260923 20:19:04.993644 21461 ts_tablet_manager.cc:1428] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260923 20:19:05.007952 21468 raft_consensus.cc:482] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:19:05.008270 21468 raft_consensus.cc:504] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } }
I20260923 20:19:05.009963 21468 leader_election.cc:288] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0bfd44eba284296873dbd9465c781f3 (127.5.83.66:42385), bd49ffff1b4b41beb52ee5ec19a552e7 (127.5.83.67:34787)
I20260923 20:19:05.017186 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:05.010410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:42497 (local address 127.5.83.67:34787)
0923 20:19:05.010705 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:05.010713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:05.011447 (+ 734us) server_negotiation.cc:407] Connection header received
0923 20:19:05.011663 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:05.011668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:05.011737 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:05.011827 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:19:05.012723 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.013444 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.015329 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.016113 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.016269 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:05.016384 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:05.016772 (+ 388us) server_negotiation.cc:299] Negotiation successful
0923 20:19:05.016901 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260923 20:19:05.017222 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:05.010487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:51471 (local address 127.5.83.66:42385)
0923 20:19:05.010681 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:05.010687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:05.011442 (+ 755us) server_negotiation.cc:407] Connection header received
0923 20:19:05.011644 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:05.011649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:05.011729 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:05.011827 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:19:05.012675 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.013482 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.015461 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.016228 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.016385 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:05.016496 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:05.016828 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 20:19:05.016964 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 20:19:05.017385 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:05.010223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:34787 (local address 127.5.83.65:42497)
0923 20:19:05.011306 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:05.011332 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:05.011537 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:05.011857 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:05.011862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:05.011908 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:19:05.012597 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:05.012613 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.013597 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.013603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:05.015163 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:05.015177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.016343 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.016349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:05.016594 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:05.016637 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 20:19:05.016745 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:19:05.016896 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":24,"spinlock_wait_cycles":30080,"thread_start_us":602,"threads_started":1}
I20260923 20:19:05.017413 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:05.010342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:42385 (local address 127.5.83.65:51471)
0923 20:19:05.011307 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:05.011332 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:05.011487 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:05.011857 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:05.011862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:05.011908 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:19:05.012451 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:05.012468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.013607 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.013614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:05.015296 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:05.015309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.016457 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.016464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:05.016676 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:05.016700 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:19:05.016804 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:19:05.016944 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":57,"spinlock_wait_cycles":11520,"thread_start_us":583,"threads_started":1}
I20260923 20:19:05.018540 21397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9120e360b57441ecb02e07e813679fe4" candidate_uuid: "6448f7afb688403d9ac2c8f667355499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" is_pre_election: true
I20260923 20:19:05.018564 21264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9120e360b57441ecb02e07e813679fe4" candidate_uuid: "6448f7afb688403d9ac2c8f667355499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfd44eba284296873dbd9465c781f3" is_pre_election: true
I20260923 20:19:05.019032 21264 raft_consensus.cc:2454] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6448f7afb688403d9ac2c8f667355499 in term 0.
I20260923 20:19:05.019033 21397 raft_consensus.cc:2454] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6448f7afb688403d9ac2c8f667355499 in term 0.
I20260923 20:19:05.019763 21066 leader_election.cc:302] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6448f7afb688403d9ac2c8f667355499, bd49ffff1b4b41beb52ee5ec19a552e7, c0bfd44eba284296873dbd9465c781f3; no voters:
I20260923 20:19:05.020176 21468 raft_consensus.cc:2781] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:19:05.020323 21468 raft_consensus.cc:482] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:19:05.020447 21468 raft_consensus.cc:3037] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:19:05.022879 21468 raft_consensus.cc:504] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } }
I20260923 20:19:05.023523 21468 leader_election.cc:288] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [CANDIDATE]: Term 1 election: Requested vote from peers c0bfd44eba284296873dbd9465c781f3 (127.5.83.66:42385), bd49ffff1b4b41beb52ee5ec19a552e7 (127.5.83.67:34787)
I20260923 20:19:05.023933 21264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9120e360b57441ecb02e07e813679fe4" candidate_uuid: "6448f7afb688403d9ac2c8f667355499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfd44eba284296873dbd9465c781f3"
I20260923 20:19:05.024008 21397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9120e360b57441ecb02e07e813679fe4" candidate_uuid: "6448f7afb688403d9ac2c8f667355499" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7"
I20260923 20:19:05.024135 21264 raft_consensus.cc:3037] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:19:05.024250 21397 raft_consensus.cc:3037] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:19:05.026598 21264 raft_consensus.cc:2454] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6448f7afb688403d9ac2c8f667355499 in term 1.
I20260923 20:19:05.026672 21397 raft_consensus.cc:2454] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6448f7afb688403d9ac2c8f667355499 in term 1.
I20260923 20:19:05.027081 21066 leader_election.cc:302] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6448f7afb688403d9ac2c8f667355499, c0bfd44eba284296873dbd9465c781f3; no voters:
I20260923 20:19:05.027438 21468 raft_consensus.cc:2781] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:19:05.027855 21468 raft_consensus.cc:686] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 1 LEADER]: Becoming Leader. State: Replica: 6448f7afb688403d9ac2c8f667355499, State: Running, Role: LEADER
I20260923 20:19:05.028314 21468 consensus_queue.cc:237] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } }
I20260923 20:19:05.033475 20984 catalog_manager.cc:5495] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6448f7afb688403d9ac2c8f667355499 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "6448f7afb688403d9ac2c8f667355499" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } health_report { overall_health: HEALTHY } } }
I20260923 20:19:05.052635 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:05.047516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.5.83.65:43323)
0923 20:19:05.047693 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:05.047698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:05.047716 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:19:05.047829 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:05.047832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:05.047940 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:05.048067 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:19:05.048429 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.049190 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.050625 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.051037 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.051125 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:05.051552 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:19:05.052080 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:19:05.052185 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:05.052293 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 20:19:05.052363 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 20:19:05.060708 21397 raft_consensus.cc:1261] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7 [term 1 FOLLOWER]: Refusing update from remote peer 6448f7afb688403d9ac2c8f667355499: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:19:05.060708 21264 raft_consensus.cc:1261] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3 [term 1 FOLLOWER]: Refusing update from remote peer 6448f7afb688403d9ac2c8f667355499: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:19:05.061865 21471 consensus_queue.cc:1035] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:19:05.062234 21468 consensus_queue.cc:1035] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:19:05.084744 21476 mvcc.cc:205] Tried to move back new op lower bound from 7332637675758129152 to 7332637675640066048. Current Snapshot: MvccSnapshot[applied={T|T < 7332637675758129152}]
I20260923 20:19:05.096949 20984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:19:05.097259 20984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:19:05.100833 20984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Sending DeleteTablet for 3 replicas of tablet 9120e360b57441ecb02e07e813679fe4
I20260923 20:19:05.102309 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 21047
I20260923 20:19:05.102202 21377 tablet_service.cc:1514] Processing DeleteTablet for tablet 9120e360b57441ecb02e07e813679fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:19:05 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:35078
I20260923 20:19:05.102209 21244 tablet_service.cc:1514] Processing DeleteTablet for tablet 9120e360b57441ecb02e07e813679fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:19:05 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:35608
I20260923 20:19:05.103719 21110 tablet_service.cc:1514] Processing DeleteTablet for tablet 9120e360b57441ecb02e07e813679fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:19:05 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:51308
I20260923 20:19:05.104751 21487 tablet_replica.cc:330] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499: stopping tablet replica
I20260923 20:19:05.105330 21487 raft_consensus.cc:2229] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:19:05.106238 21487 raft_consensus.cc:2258] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:19:05.108561 21487 ts_tablet_manager.cc:1905] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:19:05.120009 21487 ts_tablet_manager.cc:1918] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:19:05.120321 21487 log.cc:1196] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/9120e360b57441ecb02e07e813679fe4
Sep 23 20:19:05 dist-test-slave-vl24 krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:19:05.121021 21487 ts_tablet_manager.cc:1939] T 9120e360b57441ecb02e07e813679fe4 P 6448f7afb688403d9ac2c8f667355499: Deleting consensus metadata
I20260923 20:19:05.123581 20971 catalog_manager.cc:4841] TS 6448f7afb688403d9ac2c8f667355499 (127.5.83.65:43323): tablet 9120e360b57441ecb02e07e813679fe4 (table test-table [id=39a30e19b35c40f1aa1ed6d17922dfb6]) successfully deleted
I20260923 20:19:05.125948 21484 tablet_replica.cc:330] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7: stopping tablet replica
I20260923 20:19:05.126565 21485 tablet_replica.cc:330] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3: stopping tablet replica
I20260923 20:19:05.127188 21485 raft_consensus.cc:2229] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:19:05.127737 21485 raft_consensus.cc:2258] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:19:05.128226 21484 raft_consensus.cc:2229] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:19:05.128741 21484 raft_consensus.cc:2258] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:19:05.130760 21485 ts_tablet_manager.cc:1905] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:19:05.130756 21484 ts_tablet_manager.cc:1905] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:19:05.134272 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:05.104643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.5.83.65:43323)
0923 20:19:05.104805 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:05.104810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:05.104828 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:19:05.104884 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:05.104889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:05.104969 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:05.105070 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:05.108954 (+ 3884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.110206 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.114349 (+ 4143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.115154 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.115279 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:05.126555 (+ 11276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:05.126566 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:05.126581 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:05.126624 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:05.130417 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:05.131441 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:05.131450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:05.131453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:05.131646 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:05.132148 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:05.132155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:05.132160 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:05.132724 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:05.133008 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:05.133653 (+ 645us) server_negotiation.cc:299] Negotiation successful
0923 20:19:05.133903 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 20:19:05.135902 21170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:19:05.137127 21485 ts_tablet_manager.cc:1918] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:19:05.137300 21485 log.cc:1196] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/9120e360b57441ecb02e07e813679fe4
I20260923 20:19:05.137781 21485 ts_tablet_manager.cc:1939] T 9120e360b57441ecb02e07e813679fe4 P c0bfd44eba284296873dbd9465c781f3: Deleting consensus metadata
I20260923 20:19:05.139364 20973 catalog_manager.cc:4841] TS c0bfd44eba284296873dbd9465c781f3 (127.5.83.66:42385): tablet 9120e360b57441ecb02e07e813679fe4 (table test-table [id=39a30e19b35c40f1aa1ed6d17922dfb6]) successfully deleted
I20260923 20:19:05.139366 21484 ts_tablet_manager.cc:1918] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:19:05.139597 21484 log.cc:1196] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/9120e360b57441ecb02e07e813679fe4
I20260923 20:19:05.140219 21484 ts_tablet_manager.cc:1939] T 9120e360b57441ecb02e07e813679fe4 P bd49ffff1b4b41beb52ee5ec19a552e7: Deleting consensus metadata
I20260923 20:19:05.142282 20971 catalog_manager.cc:4841] TS bd49ffff1b4b41beb52ee5ec19a552e7 (127.5.83.67:34787): tablet 9120e360b57441ecb02e07e813679fe4 (table test-table [id=39a30e19b35c40f1aa1ed6d17922dfb6]) successfully deleted
I20260923 20:19:05.250247 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 21047
I20260923 20:19:05.269721 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 21181
Sep 23 20:19:05 dist-test-slave-vl24 krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:19:05.285421 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:05.270501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.5.83.66:42385)
0923 20:19:05.270709 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:05.270715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:05.270809 (+ 94us) server_negotiation.cc:407] Connection header received
0923 20:19:05.271022 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:05.271027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:05.271100 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:05.271209 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:05.272502 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.273290 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.274836 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.275227 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.275361 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:05.279725 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:05.279740 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:05.279756 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:05.279790 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:05.282490 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:05.283192 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:05.283199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:05.283202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:05.283323 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:05.283706 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:05.283712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:05.283715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:05.284109 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:05.284424 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:05.284856 (+ 432us) server_negotiation.cc:299] Negotiation successful
0923 20:19:05.285096 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:19:05.286058 21304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:19:05.377058 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 21181
I20260923 20:19:05.395164 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 21314
Sep 23 20:19:05 dist-test-slave-vl24 krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:19:05.411553 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:05.396047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.5.83.67:34787)
0923 20:19:05.396244 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:05.396250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:05.396284 (+ 34us) server_negotiation.cc:407] Connection header received
0923 20:19:05.396352 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:05.396358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:05.396438 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:05.396607 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:05.397994 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.398777 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.400176 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:05.400691 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:05.400835 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:05.405420 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:05.405437 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:05.405454 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:05.405498 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:05.408298 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:05.409304 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:05.409314 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:05.409319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:05.409478 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:05.409892 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:05.409903 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:05.409907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:05.410265 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:05.410593 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:05.410944 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 20:19:05.411193 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:19:05.412348 21437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:19:05.503571 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 21314
I20260923 20:19:05.520802 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 20954
I20260923 20:19:05.522082 21016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:19:05.633319 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 20954
I20260923 20:19:25.656035 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:46799
--webserver_interface=127.5.83.126
--webserver_port=42513
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:46799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:19:25.995430 21492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:19:25.995793 21492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:19:25.995957 21492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:19:26.003455 21492 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 20:19:26.003607 21492 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 20:19:26.003701 21492 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 20:19:26.003793 21492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:19:26.003892 21492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:19:26.004019 21492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:19:26.004103 21492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:19:26.004192 21492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:19:26.014492 21492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:46799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:46799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=42513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:19:26.015892 21492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:19:26.017446 21492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:19:26.042196 21501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:26.042164 21498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:26.042248 21499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:19:26.043394 21492 server_base.cc:1034] running on GCE node
I20260923 20:19:26.044309 21492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:19:26.046036 21492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:19:26.047333 21492 hybrid_clock.cc:648] HybridClock initialized: now 1790194766047248 us; error 48 us; skew 500 ppm
I20260923 20:19:26.047771 21492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:19:26 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194766, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:19:26.051796 21492 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:19:26.053679 21492 webserver.cc:460] Webserver started at http://127.5.83.126:42513/ using document root <none> and password file <none>
I20260923 20:19:26.054476 21492 fs_manager.cc:359] Metadata directory not provided
I20260923 20:19:26.054584 21492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:19:26.061089 21492 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 20:19:26.064296 21508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:26.065917 21492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:19:26.066071 21492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "e277bfca93634e4fbd79f266c2193cb7"
format_stamp: "Formatted at 2026-09-23 20:19:03 on dist-test-slave-vl24"
server_key: "4b2ed2c783707f3e3092ff1f66466f7c"
server_key_iv: "182b616e179a9d36483cc0881c51da7c"
server_key_version: "encryptionkey@0"
I20260923 20:19:26.067157 21492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:19:26.088162 21492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:19:26.093706 21492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:19:26.094010 21492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:19:26.114045 21492 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:46799
I20260923 20:19:26.114090 21559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:46799 every 8 connection(s)
I20260923 20:19:26.115795 21492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:19:26.123365 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 21492
I20260923 20:19:26.124334 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:43323
--local_ip_for_outbound_sockets=127.5.83.65
--tserver_master_addrs=127.5.83.126:46799
--webserver_port=42753
--webserver_interface=127.5.83.65
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:19:26.124641 21560 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:19:26.129170 21560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Bootstrap starting.
I20260923 20:19:26.151333 21560 log.cc:824] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Log is configured to *not* fsync() on all Append() calls
I20260923 20:19:26.165105 21560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:19:26.165791 21560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Bootstrap complete.
I20260923 20:19:26.176836 21560 raft_consensus.cc:348] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } }
I20260923 20:19:26.177877 21560 raft_consensus.cc:729] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e277bfca93634e4fbd79f266c2193cb7, State: Initialized, Role: FOLLOWER
I20260923 20:19:26.178593 21560 consensus_queue.cc:260] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } }
I20260923 20:19:26.178894 21560 raft_consensus.cc:388] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:19:26.179033 21560 raft_consensus.cc:482] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:19:26.179229 21560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 1 FOLLOWER]: Advancing to term 2
I20260923 20:19:26.183693 21560 raft_consensus.cc:504] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } }
I20260923 20:19:26.184324 21560 leader_election.cc:302] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e277bfca93634e4fbd79f266c2193cb7; no voters:
I20260923 20:19:26.185107 21560 leader_election.cc:288] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 20:19:26.185263 21564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 20:19:26.186606 21564 raft_consensus.cc:686] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 2 LEADER]: Becoming Leader. State: Replica: e277bfca93634e4fbd79f266c2193cb7, State: Running, Role: LEADER
I20260923 20:19:26.187337 21564 consensus_queue.cc:237] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } }
I20260923 20:19:26.187703 21560 sys_catalog.cc:564] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:19:26.190508 21566 sys_catalog.cc:455] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e277bfca93634e4fbd79f266c2193cb7. Latest consensus state: current_term: 2 leader_uuid: "e277bfca93634e4fbd79f266c2193cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } } }
I20260923 20:19:26.190505 21565 sys_catalog.cc:455] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e277bfca93634e4fbd79f266c2193cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } } }
I20260923 20:19:26.191116 21566 sys_catalog.cc:458] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: This master's current role is: LEADER
I20260923 20:19:26.191192 21565 sys_catalog.cc:458] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: This master's current role is: LEADER
I20260923 20:19:26.192170 21570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:19:26.208612 21570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:19:26.210340 21570 catalog_manager.cc:1259] Loaded cluster ID: aa81963a24b94261a8e6fea57ff2f34a
I20260923 20:19:26.210456 21570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:19:26.214969 21570 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:19:26.216802 21570 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Loaded TSK: 0
I20260923 20:19:26.274489 21570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Generated new TSK 1
I20260923 20:19:26.274938 21570 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:19:26.513926 21562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:19:26.514228 21562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:19:26.514385 21562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:19:26.522219 21562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:19:26.522464 21562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:19:26.522667 21562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:19:26.533310 21562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:43323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=42753
--tserver_master_addrs=127.5.83.126:46799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:19:26.534685 21562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:19:26.536095 21562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:19:26.551167 21590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:26.551198 21588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:26.551451 21587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:19:26.552106 21562 server_base.cc:1034] running on GCE node
I20260923 20:19:26.552883 21562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:19:26.554013 21562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:19:26.555246 21562 hybrid_clock.cc:648] HybridClock initialized: now 1790194766555203 us; error 36 us; skew 500 ppm
I20260923 20:19:26.555610 21562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:19:26 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194766, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:19:26.558993 21562 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:19:26.560671 21562 webserver.cc:460] Webserver started at http://127.5.83.65:42753/ using document root <none> and password file <none>
I20260923 20:19:26.561460 21562 fs_manager.cc:359] Metadata directory not provided
I20260923 20:19:26.561607 21562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:19:26.567399 21562 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 20:19:26.570760 21597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:26.572396 21562 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 20:19:26.572618 21562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "6448f7afb688403d9ac2c8f667355499"
format_stamp: "Formatted at 2026-09-23 20:19:03 on dist-test-slave-vl24"
server_key: "424d1305d2e83e904fa29f1d1c1b8522"
server_key_iv: "b2b438ffc6a3d23404b3ffea1f22fb67"
server_key_version: "encryptionkey@0"
I20260923 20:19:26.573666 21562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:19:26.602478 21562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:19:26.607069 21562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:19:26.607371 21562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:19:26.608405 21562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:19:26.609783 21562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:19:26.609911 21562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:26.610100 21562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:19:26.610257 21562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:26.644311 21562 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:43323
I20260923 20:19:26.644400 21709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:43323 every 8 connection(s)
I20260923 20:19:26.645799 21562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:19:26.653291 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 21562
I20260923 20:19:26.654105 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:42385
--local_ip_for_outbound_sockets=127.5.83.66
--tserver_master_addrs=127.5.83.126:46799
--webserver_port=40207
--webserver_interface=127.5.83.66
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:19:26 dist-test-slave-vl24 krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194766, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:19:26.673050 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:26.648741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:39755 (local address 127.5.83.126:46799)
0923 20:19:26.649491 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:26.649508 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:26.649864 (+ 356us) server_negotiation.cc:407] Connection header received
0923 20:19:26.651901 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:26.651915 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:26.652273 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:26.652633 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:26.654755 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:26.656109 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:26.658639 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:26.659312 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:26.659457 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:26.664022 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:26.664030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:26.664046 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:26.664089 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:26.667193 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:26.668353 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:26.668361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:26.668366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:26.668613 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:26.669147 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:26.669150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:26.669152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:26.669851 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:26.670103 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:26.671219 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0923 20:19:26.671770 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":290,"threads_started":1}
I20260923 20:19:26.673152 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:26.648855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:39755)
0923 20:19:26.649693 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:26.649731 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:26.651394 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:26.652854 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:26.652866 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:26.653651 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:19:26.654451 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:26.654479 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:26.656302 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:26.656310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:26.658483 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:26.658502 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:26.659555 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:26.659565 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:26.659890 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:26.660685 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:19:26.660720 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:19:26.663653 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:19:26.667398 (+ 3745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:19:26.667412 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:19:26.667433 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:19:26.668181 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:19:26.668752 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:19:26.668759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:19:26.668766 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:19:26.669031 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:19:26.670124 (+ 1093us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:19:26.670136 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:19:26.670617 (+ 481us) client_negotiation.cc:769] Sending connection context
0923 20:19:26.670996 (+ 379us) client_negotiation.cc:240] Negotiation successful
0923 20:19:26.671677 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":395,"threads_started":1}
I20260923 20:19:26.676468 21710 heartbeater.cc:347] Connected to a master server at 127.5.83.126:46799
I20260923 20:19:26.676937 21710 heartbeater.cc:464] Registering TS with master...
I20260923 20:19:26.677807 21710 heartbeater.cc:511] Master 127.5.83.126:46799 requested a full tablet report, sending...
I20260923 20:19:26.680375 21525 ts_manager.cc:194] Registered new tserver with Master: 6448f7afb688403d9ac2c8f667355499 (127.5.83.65:43323)
I20260923 20:19:26.682778 21525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:39755
W20260923 20:19:27.005887 21714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:19:27.006193 21714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:19:27.006302 21714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:19:27.013804 21714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:19:27.013937 21714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:19:27.014060 21714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:19:27.024315 21714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:42385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=40207
--tserver_master_addrs=127.5.83.126:46799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:19:27.025717 21714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:19:27.027065 21714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:19:27.041074 21720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:27.041210 21722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:27.041229 21719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:19:27.042223 21714 server_base.cc:1034] running on GCE node
I20260923 20:19:27.042831 21714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:19:27.043812 21714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:19:27.045034 21714 hybrid_clock.cc:648] HybridClock initialized: now 1790194767044971 us; error 55 us; skew 500 ppm
I20260923 20:19:27.045339 21714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:19:27 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194767, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:19:27.048468 21714 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:19:27.050027 21714 webserver.cc:460] Webserver started at http://127.5.83.66:40207/ using document root <none> and password file <none>
I20260923 20:19:27.050648 21714 fs_manager.cc:359] Metadata directory not provided
I20260923 20:19:27.050757 21714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:19:27.056069 21714 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260923 20:19:27.059026 21729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:27.060518 21714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:19:27.060781 21714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "c0bfd44eba284296873dbd9465c781f3"
format_stamp: "Formatted at 2026-09-23 20:19:04 on dist-test-slave-vl24"
server_key: "67d5abec5c07b0ed8cfdd9abb97b5e44"
server_key_iv: "b20c2140cc78cd468e0f964c7bd7fdc2"
server_key_version: "encryptionkey@0"
I20260923 20:19:27.061795 21714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:19:27.093633 21714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:19:27.097999 21714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:19:27.098281 21714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:19:27.099308 21714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:19:27.100709 21714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:19:27.100795 21714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:27.100945 21714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:19:27.101027 21714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:27.135854 21714 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:42385
I20260923 20:19:27.135948 21841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:42385 every 8 connection(s)
I20260923 20:19:27.137492 21714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:19:27.146080 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 21714
I20260923 20:19:27.146826 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:34787
--local_ip_for_outbound_sockets=127.5.83.67
--tserver_master_addrs=127.5.83.126:46799
--webserver_port=38517
--webserver_interface=127.5.83.67
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 20:19:27 dist-test-slave-vl24 krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194767, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:19:27.159739 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.140154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:47455 (local address 127.5.83.126:46799)
0923 20:19:27.140345 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:27.140350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:27.141728 (+ 1378us) server_negotiation.cc:407] Connection header received
0923 20:19:27.143626 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:27.143632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:27.143762 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:27.143877 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:27.145526 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.146631 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.148490 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.149306 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.149452 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.153097 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:27.153105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:27.153113 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:27.153159 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:27.156353 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:27.157391 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:27.157395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:27.157398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:27.157522 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:27.158017 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:27.158020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:27.158022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:27.158284 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:27.158391 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:27.159305 (+ 914us) server_negotiation.cc:299] Negotiation successful
0923 20:19:27.159490 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:19:27.161423 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.140789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:47455)
0923 20:19:27.141545 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:27.141589 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:27.143335 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:27.144139 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:27.144148 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:27.144774 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:19:27.145341 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.145358 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.146859 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.146864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.148315 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.148329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.149597 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.149602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.149840 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.150554 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:19:27.150577 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:19:27.152853 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:19:27.156548 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:19:27.156560 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:19:27.156581 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:19:27.157264 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:19:27.157629 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:19:27.157637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:19:27.157644 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:19:27.157908 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:19:27.158405 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:19:27.158417 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:19:27.158920 (+ 503us) client_negotiation.cc:769] Sending connection context
0923 20:19:27.159237 (+ 317us) client_negotiation.cc:240] Negotiation successful
0923 20:19:27.159877 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":310,"threads_started":1}
I20260923 20:19:27.167135 21842 heartbeater.cc:347] Connected to a master server at 127.5.83.126:46799
I20260923 20:19:27.167493 21842 heartbeater.cc:464] Registering TS with master...
I20260923 20:19:27.168468 21842 heartbeater.cc:511] Master 127.5.83.126:46799 requested a full tablet report, sending...
I20260923 20:19:27.170589 21525 ts_manager.cc:194] Registered new tserver with Master: c0bfd44eba284296873dbd9465c781f3 (127.5.83.66:42385)
I20260923 20:19:27.172019 21525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:47455
W20260923 20:19:27.490785 21845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:19:27.491065 21845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:19:27.491156 21845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:19:27.498433 21845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:19:27.498538 21845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:19:27.498646 21845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:19:27.508447 21845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:34787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=38517
--tserver_master_addrs=127.5.83.126:46799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:19:27.509738 21845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:19:27.510993 21845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:19:27.524125 21851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:27.524152 21853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:27.524421 21850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:19:27.525151 21845 server_base.cc:1034] running on GCE node
I20260923 20:19:27.525697 21845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:19:27.526568 21845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:19:27.527762 21845 hybrid_clock.cc:648] HybridClock initialized: now 1790194767527704 us; error 47 us; skew 500 ppm
I20260923 20:19:27.528079 21845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:19:27 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194767, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:19:27.531219 21845 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:19:27.532775 21845 webserver.cc:460] Webserver started at http://127.5.83.67:38517/ using document root <none> and password file <none>
I20260923 20:19:27.533458 21845 fs_manager.cc:359] Metadata directory not provided
I20260923 20:19:27.533628 21845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:19:27.539258 21845 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260923 20:19:27.542078 21860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:27.543390 21845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:19:27.543581 21845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "bd49ffff1b4b41beb52ee5ec19a552e7"
format_stamp: "Formatted at 2026-09-23 20:19:04 on dist-test-slave-vl24"
server_key: "92453e8fe57b539b0859c0958314d55a"
server_key_iv: "2ca4da513c0e7e0255c7e4d382555fcb"
server_key_version: "encryptionkey@0"
I20260923 20:19:27.544601 21845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:19:27.559688 21845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:19:27.564021 21845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:19:27.564301 21845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:19:27.565428 21845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:19:27.566874 21845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:19:27.567001 21845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:27.567137 21845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:19:27.567224 21845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:27.604458 21845 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:34787
I20260923 20:19:27.604594 21972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:34787 every 8 connection(s)
I20260923 20:19:27.605944 21845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:19:27.606520 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 21845
Sep 23 20:19:27 dist-test-slave-vl24 krb5kdc[20941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194767, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:19:27.622591 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.607805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.5.83.126:46799)
0923 20:19:27.608084 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:27.608091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:27.608472 (+ 381us) server_negotiation.cc:407] Connection header received
0923 20:19:27.608741 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:27.608749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:27.608847 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:27.608972 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:27.611089 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.611856 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.613291 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.613725 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.613863 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.616621 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:27.616627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:27.616632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:27.616675 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:27.619713 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:27.620384 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:27.620390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:27.620393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:27.620569 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:27.621069 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:27.621074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:27.621078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:27.621371 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:27.621646 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:27.622006 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 20:19:27.622233 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:19:27.629736 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.608864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:47233 (local address 127.5.83.126:46799)
0923 20:19:27.611129 (+ 2265us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:27.611134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:27.611158 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:19:27.613777 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:27.613782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:27.613869 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:27.614002 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:27.615580 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.616358 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.618248 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.619004 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.619155 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.623130 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:27.623135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:27.623138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:27.623168 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:27.626342 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:27.627150 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:27.627161 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:27.627163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:27.627286 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:27.627770 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:27.627777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:27.627786 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:27.628070 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:27.628235 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:27.629040 (+ 805us) server_negotiation.cc:299] Negotiation successful
0923 20:19:27.629291 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2158,"thread_start_us":2032,"threads_started":1}
I20260923 20:19:27.630693 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.609425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:47233)
0923 20:19:27.610516 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:27.610577 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:27.613500 (+ 2923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:27.614265 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:27.614274 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:27.614812 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:19:27.615383 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.615401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.616579 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.616590 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.618089 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.618107 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.619260 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.619266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.619469 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.620120 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:19:27.620144 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:19:27.622846 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:19:27.626499 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:19:27.626509 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:19:27.626528 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:19:27.627005 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:19:27.627409 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:19:27.627416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:19:27.627420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:19:27.627630 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:19:27.628209 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:19:27.628218 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:19:27.628647 (+ 429us) client_negotiation.cc:769] Sending connection context
0923 20:19:27.628945 (+ 298us) client_negotiation.cc:240] Negotiation successful
0923 20:19:27.629498 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":412,"threads_started":1}
I20260923 20:19:27.632112 21973 heartbeater.cc:347] Connected to a master server at 127.5.83.126:46799
I20260923 20:19:27.632428 21973 heartbeater.cc:464] Registering TS with master...
I20260923 20:19:27.633191 21973 heartbeater.cc:511] Master 127.5.83.126:46799 requested a full tablet report, sending...
I20260923 20:19:27.634604 21525 ts_manager.cc:194] Registered new tserver with Master: bd49ffff1b4b41beb52ee5ec19a552e7 (127.5.83.67:34787)
I20260923 20:19:27.635409 21525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:47233
I20260923 20:19:27.637950 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:19:27.644449 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.638903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.5.83.126:46799)
0923 20:19:27.639106 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:27.639111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:27.639542 (+ 431us) server_negotiation.cc:407] Connection header received
0923 20:19:27.639714 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:27.639718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:27.639853 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:27.639983 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:19:27.640346 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.641103 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.642591 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.642986 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.643079 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.643512 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:19:27.643793 (+ 281us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 20:19:27.644214 (+ 421us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54112: token expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 20:19:27.644593 21976 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54112: token expired
W20260923 20:19:27.645072 21446 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.83.126:46799)
I20260923 20:19:27.658212 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.645643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.5.83.126:46799)
0923 20:19:27.645791 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:27.645796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:27.645813 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:19:27.645951 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:27.645957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:27.646036 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:27.646140 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:27.647438 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.648578 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.650110 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.650488 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.650584 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.653277 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:27.653281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:27.653284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:27.653317 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:27.655836 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:27.656442 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:27.656446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:27.656448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:27.656630 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:27.656988 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:27.656991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:27.656993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:27.657285 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:27.657479 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:27.657723 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 20:19:27.657878 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:19:27.661880 21525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:19:27.669703 21525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:19:27.686424 21710 heartbeater.cc:503] Master 127.5.83.126:46799 was elected leader, sending a full tablet report...
I20260923 20:19:27.693630 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.681964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:42385 (local address 127.0.0.1:52556)
0923 20:19:27.683615 (+ 1651us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:27.683629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:27.683764 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:27.684433 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:27.684441 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:27.684483 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:19:27.685571 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.685584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.687518 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.687524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.690577 (+ 3053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.690597 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.692513 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.692519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.692805 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.692845 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:19:27.692912 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 20:19:27.693050 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1539,"mutex_wait_us":144,"thread_start_us":1406,"threads_started":1}
I20260923 20:19:27.693651 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.681297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43323 (local address 127.0.0.1:43968)
0923 20:19:27.682286 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:27.682331 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:27.682504 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:27.684901 (+ 2397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:27.684908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:27.684937 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:19:27.685381 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.685395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.687535 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.687538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.690617 (+ 3079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.690632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.692593 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.692605 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.692740 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.692784 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 20:19:27.692877 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 20:19:27.693007 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":243,"thread_start_us":558,"threads_started":1}
I20260923 20:19:27.694123 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.681700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.5.83.65:43323)
0923 20:19:27.684309 (+ 2609us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:27.684319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:27.684347 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:19:27.684437 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:27.684443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:27.684683 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:27.685089 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:19:27.685798 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.687355 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.690768 (+ 3413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.692238 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.693278 (+ 1040us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.693579 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:27.693651 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 20:19:27.693818 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2438,"thread_start_us":2306,"threads_started":1}
I20260923 20:19:27.694123 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.683548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.5.83.67:34787)
0923 20:19:27.684713 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:27.684724 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:27.685536 (+ 812us) server_negotiation.cc:407] Connection header received
0923 20:19:27.685648 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:27.685655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:27.685874 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:27.686071 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:19:27.687099 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.688835 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.690864 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.692234 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.692372 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.692649 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:27.693178 (+ 529us) server_negotiation.cc:299] Negotiation successful
0923 20:19:27.693341 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":964,"thread_start_us":798,"threads_started":1}
I20260923 20:19:27.694216 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.682892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:34787 (local address 127.0.0.1:41830)
0923 20:19:27.685262 (+ 2370us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:27.685277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:27.685411 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:27.686071 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:27.686075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:27.686097 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:19:27.686472 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.686484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.689014 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.689018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.690708 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.690725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.692807 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.692813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.692983 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.693013 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:19:27.693094 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 20:19:27.693234 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2238,"mutex_wait_us":74,"thread_start_us":2046,"threads_started":1}
I20260923 20:19:27.695004 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.682486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52556 (local address 127.5.83.66:42385)
0923 20:19:27.683166 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:27.683178 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:27.683730 (+ 552us) server_negotiation.cc:407] Connection header received
0923 20:19:27.683930 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:27.683940 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:27.684196 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:27.685359 (+ 1163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:19:27.685734 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.687344 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.690773 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.692234 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.694137 (+ 1903us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.694480 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:27.694531 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 20:19:27.694714 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":399,"threads_started":1}
I20260923 20:19:27.698087 21645 tablet_service.cc:1467] Processing CreateTablet for tablet 4c52bc43039145e98fff473e73c229aa (DEFAULT_TABLE table=test-table [id=2d7c1f12c8994269abeff3e05d1ec8ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:19:27.698087 21908 tablet_service.cc:1467] Processing CreateTablet for tablet 4c52bc43039145e98fff473e73c229aa (DEFAULT_TABLE table=test-table [id=2d7c1f12c8994269abeff3e05d1ec8ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:19:27.699611 21908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c52bc43039145e98fff473e73c229aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:19:27.699622 21645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c52bc43039145e98fff473e73c229aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:19:27.702936 21777 tablet_service.cc:1467] Processing CreateTablet for tablet 4c52bc43039145e98fff473e73c229aa (DEFAULT_TABLE table=test-table [id=2d7c1f12c8994269abeff3e05d1ec8ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:19:27.704120 21777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c52bc43039145e98fff473e73c229aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:19:27.715976 21990 tablet_bootstrap.cc:492] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7: Bootstrap starting.
I20260923 20:19:27.721024 21990 tablet_bootstrap.cc:654] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7: Neither blocks nor log segments found. Creating new log.
I20260923 20:19:27.721534 21989 tablet_bootstrap.cc:492] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: Bootstrap starting.
I20260923 20:19:27.722908 21990 log.cc:824] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7: Log is configured to *not* fsync() on all Append() calls
I20260923 20:19:27.725039 21989 tablet_bootstrap.cc:654] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: Neither blocks nor log segments found. Creating new log.
I20260923 20:19:27.725066 21991 tablet_bootstrap.cc:492] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3: Bootstrap starting.
I20260923 20:19:27.726289 21989 log.cc:824] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: Log is configured to *not* fsync() on all Append() calls
I20260923 20:19:27.727037 21990 tablet_bootstrap.cc:492] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7: No bootstrap required, opened a new log
I20260923 20:19:27.727335 21990 ts_tablet_manager.cc:1397] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260923 20:19:27.728942 21989 tablet_bootstrap.cc:492] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: No bootstrap required, opened a new log
I20260923 20:19:27.729207 21989 ts_tablet_manager.cc:1397] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260923 20:19:27.729197 21991 tablet_bootstrap.cc:654] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3: Neither blocks nor log segments found. Creating new log.
I20260923 20:19:27.730460 21991 log.cc:824] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3: Log is configured to *not* fsync() on all Append() calls
I20260923 20:19:27.733604 21991 tablet_bootstrap.cc:492] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3: No bootstrap required, opened a new log
I20260923 20:19:27.733865 21991 ts_tablet_manager.cc:1397] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260923 20:19:27.737397 21989 raft_consensus.cc:348] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } }
I20260923 20:19:27.737784 21989 raft_consensus.cc:374] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:19:27.737903 21989 raft_consensus.cc:729] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6448f7afb688403d9ac2c8f667355499, State: Initialized, Role: FOLLOWER
I20260923 20:19:27.738258 21990 raft_consensus.cc:348] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } }
I20260923 20:19:27.738695 21990 raft_consensus.cc:374] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:19:27.738538 21989 consensus_queue.cc:260] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } }
I20260923 20:19:27.738790 21990 raft_consensus.cc:729] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd49ffff1b4b41beb52ee5ec19a552e7, State: Initialized, Role: FOLLOWER
I20260923 20:19:27.739387 21990 consensus_queue.cc:260] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } }
I20260923 20:19:27.740620 21989 ts_tablet_manager.cc:1428] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260923 20:19:27.741590 21973 heartbeater.cc:503] Master 127.5.83.126:46799 was elected leader, sending a full tablet report...
I20260923 20:19:27.743429 21990 ts_tablet_manager.cc:1428] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260923 20:19:27.744937 21991 raft_consensus.cc:348] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } }
I20260923 20:19:27.745437 21991 raft_consensus.cc:374] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:19:27.745541 21991 raft_consensus.cc:729] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bfd44eba284296873dbd9465c781f3, State: Initialized, Role: FOLLOWER
I20260923 20:19:27.746556 21991 consensus_queue.cc:260] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } }
I20260923 20:19:27.747685 21842 heartbeater.cc:503] Master 127.5.83.126:46799 was elected leader, sending a full tablet report...
I20260923 20:19:27.749477 21991 ts_tablet_manager.cc:1428] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
W20260923 20:19:27.858717 21974 tablet.cc:2369] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:19:27.876103 21996 raft_consensus.cc:482] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:19:27.876360 21996 raft_consensus.cc:504] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } }
I20260923 20:19:27.878185 21996 leader_election.cc:288] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6448f7afb688403d9ac2c8f667355499 (127.5.83.65:43323), c0bfd44eba284296873dbd9465c781f3 (127.5.83.66:42385)
I20260923 20:19:27.885456 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.878781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:38573 (local address 127.5.83.65:43323)
0923 20:19:27.879028 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:27.879037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:27.879062 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:19:27.879145 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:27.879150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:27.879229 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:27.879335 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:19:27.879833 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.880899 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.883374 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.884601 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.884739 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.884844 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:27.885111 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 20:19:27.885203 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:19:27.885687 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.878565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:42385 (local address 127.5.83.67:51551)
0923 20:19:27.879379 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:27.879394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:27.879513 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:27.879824 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:27.879830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:27.879855 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:19:27.880115 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.880122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.881377 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.881388 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.883425 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.883435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.884998 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.885002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.885162 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.885190 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 20:19:27.885267 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 20:19:27.885377 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":535,"threads_started":1}
I20260923 20:19:27.885777 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.878457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43323 (local address 127.5.83.67:38573)
0923 20:19:27.878690 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:27.878714 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:19:27.878850 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:19:27.879378 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:19:27.879384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:19:27.879418 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:19:27.879683 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.879693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.881040 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.881044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.883210 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:19:27.883224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.884764 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.884768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:19:27.884929 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.884974 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 20:19:27.885078 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:19:27.885218 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":101,"spinlock_wait_cycles":20352}
I20260923 20:19:27.885841 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.878787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:51551 (local address 127.5.83.66:42385)
0923 20:19:27.879011 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:27.879020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:27.879466 (+ 446us) server_negotiation.cc:407] Connection header received
0923 20:19:27.879638 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:27.879641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:27.879697 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:27.879797 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:19:27.880276 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.881251 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.883563 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.884817 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.884938 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.885001 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:27.885307 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 20:19:27.885425 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:19:27.886881 21797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c52bc43039145e98fff473e73c229aa" candidate_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfd44eba284296873dbd9465c781f3" is_pre_election: true
I20260923 20:19:27.886915 21665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c52bc43039145e98fff473e73c229aa" candidate_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6448f7afb688403d9ac2c8f667355499" is_pre_election: true
I20260923 20:19:27.887408 21665 raft_consensus.cc:2454] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd49ffff1b4b41beb52ee5ec19a552e7 in term 0.
I20260923 20:19:27.887440 21797 raft_consensus.cc:2454] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd49ffff1b4b41beb52ee5ec19a552e7 in term 0.
I20260923 20:19:27.888197 21864 leader_election.cc:302] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6448f7afb688403d9ac2c8f667355499, bd49ffff1b4b41beb52ee5ec19a552e7, c0bfd44eba284296873dbd9465c781f3; no voters:
I20260923 20:19:27.888696 21996 raft_consensus.cc:2781] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:19:27.888921 21996 raft_consensus.cc:482] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:19:27.889060 21996 raft_consensus.cc:3037] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 0 FOLLOWER]: Advancing to term 1
W20260923 20:19:27.889432 21843 tablet.cc:2369] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:19:27.891443 21996 raft_consensus.cc:504] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } }
I20260923 20:19:27.892172 21996 leader_election.cc:288] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [CANDIDATE]: Term 1 election: Requested vote from peers 6448f7afb688403d9ac2c8f667355499 (127.5.83.65:43323), c0bfd44eba284296873dbd9465c781f3 (127.5.83.66:42385)
I20260923 20:19:27.892685 21797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c52bc43039145e98fff473e73c229aa" candidate_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0bfd44eba284296873dbd9465c781f3"
I20260923 20:19:27.892693 21665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c52bc43039145e98fff473e73c229aa" candidate_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6448f7afb688403d9ac2c8f667355499"
I20260923 20:19:27.892913 21797 raft_consensus.cc:3037] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:19:27.892962 21665 raft_consensus.cc:3037] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:19:27.895208 21797 raft_consensus.cc:2454] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd49ffff1b4b41beb52ee5ec19a552e7 in term 1.
I20260923 20:19:27.895519 21665 raft_consensus.cc:2454] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd49ffff1b4b41beb52ee5ec19a552e7 in term 1.
I20260923 20:19:27.895620 21864 leader_election.cc:302] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd49ffff1b4b41beb52ee5ec19a552e7, c0bfd44eba284296873dbd9465c781f3; no voters:
I20260923 20:19:27.895994 21996 raft_consensus.cc:2781] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:19:27.896425 21996 raft_consensus.cc:686] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 1 LEADER]: Becoming Leader. State: Replica: bd49ffff1b4b41beb52ee5ec19a552e7, State: Running, Role: LEADER
I20260923 20:19:27.896984 21996 consensus_queue.cc:237] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } } peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } }
W20260923 20:19:27.898108 21711 tablet.cc:2369] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:19:27.902079 21525 catalog_manager.cc:5495] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd49ffff1b4b41beb52ee5ec19a552e7 (127.5.83.67). New cstate: current_term: 1 leader_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd49ffff1b4b41beb52ee5ec19a552e7" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 34787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 } health_report { overall_health: UNKNOWN } } }
I20260923 20:19:27.933485 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.927903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.5.83.67:34787)
0923 20:19:27.928135 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:27.928141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:27.928160 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:19:27.928388 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:27.928392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:27.928473 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:27.928690 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:19:27.929083 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.929806 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.931226 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.931634 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.931770 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.932334 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:19:27.932904 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:19:27.933075 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:27.933132 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 20:19:27.933212 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 20:19:27.941524 21797 raft_consensus.cc:1261] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3 [term 1 FOLLOWER]: Refusing update from remote peer bd49ffff1b4b41beb52ee5ec19a552e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:19:27.941542 21665 raft_consensus.cc:1261] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499 [term 1 FOLLOWER]: Refusing update from remote peer bd49ffff1b4b41beb52ee5ec19a552e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:19:27.942579 21999 consensus_queue.cc:1035] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0bfd44eba284296873dbd9465c781f3" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 42385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:19:27.942970 21996 consensus_queue.cc:1035] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6448f7afb688403d9ac2c8f667355499" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 43323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:19:27.955729 22004 mvcc.cc:205] Tried to move back new op lower bound from 7332637769478746112 to 7332637769310429184. Current Snapshot: MvccSnapshot[applied={T|T < 7332637769478746112}]
I20260923 20:19:27.963022 22002 mvcc.cc:205] Tried to move back new op lower bound from 7332637769478746112 to 7332637769310429184. Current Snapshot: MvccSnapshot[applied={T|T < 7332637769478746112 or (T in {7332637769478746112})}]
I20260923 20:19:27.969856 21524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:19:27.970180 21524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:19:27.974116 21524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Sending DeleteTablet for 3 replicas of tablet 4c52bc43039145e98fff473e73c229aa
I20260923 20:19:27.975296 21645 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c52bc43039145e98fff473e73c229aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:19:27 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:43968
I20260923 20:19:27.975414 21777 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c52bc43039145e98fff473e73c229aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:19:27 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:52556
I20260923 20:19:27.975473 21908 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c52bc43039145e98fff473e73c229aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:19:27 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:41830
I20260923 20:19:27.976413 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 21562
I20260923 20:19:27.987036 22013 tablet_replica.cc:330] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3: stopping tablet replica
I20260923 20:19:27.987192 22012 tablet_replica.cc:330] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: stopping tablet replica
I20260923 20:19:27.987586 22012 raft_consensus.cc:2229] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:19:27.987576 22013 raft_consensus.cc:2229] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:19:27.988159 22012 raft_consensus.cc:2258] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:19:27.989145 22013 raft_consensus.cc:2258] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:19:27.989686 22014 tablet_replica.cc:330] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7: stopping tablet replica
I20260923 20:19:27.990100 22014 raft_consensus.cc:2229] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:19:27.990844 22012 ts_tablet_manager.cc:1905] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:19:27.991456 22013 ts_tablet_manager.cc:1905] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:19:27.991758 22014 raft_consensus.cc:2258] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:19:27.998457 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:27.977384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.5.83.65:43323)
0923 20:19:27.977627 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:27.977636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:27.977660 (+ 24us) server_negotiation.cc:407] Connection header received
0923 20:19:27.977917 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:27.977946 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:27.978090 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:27.978254 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:27.980648 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.982215 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.985055 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:27.986005 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:27.986160 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:27.990872 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:27.990884 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:27.990899 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:27.990946 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:27.994957 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:27.995892 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:27.995902 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:27.995906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:27.996089 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:27.996573 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:27.996579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:27.996582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:27.997111 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:27.997433 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:27.997810 (+ 377us) server_negotiation.cc:299] Negotiation successful
0923 20:19:27.998057 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 20:19:27.999027 22013 ts_tablet_manager.cc:1918] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:19:27.999053 22012 ts_tablet_manager.cc:1918] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:19:27.999189 21705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:19:27.999229 22012 log.cc:1196] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/4c52bc43039145e98fff473e73c229aa
I20260923 20:19:27.999365 22013 log.cc:1196] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/4c52bc43039145e98fff473e73c229aa
I20260923 20:19:27.999987 22013 ts_tablet_manager.cc:1939] T 4c52bc43039145e98fff473e73c229aa P c0bfd44eba284296873dbd9465c781f3: Deleting consensus metadata
I20260923 20:19:28.002461 21512 catalog_manager.cc:4841] TS c0bfd44eba284296873dbd9465c781f3 (127.5.83.66:42385): tablet 4c52bc43039145e98fff473e73c229aa (table test-table [id=2d7c1f12c8994269abeff3e05d1ec8ce]) successfully deleted
I20260923 20:19:28.006127 22014 ts_tablet_manager.cc:1905] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:19:28.014545 22014 ts_tablet_manager.cc:1918] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:19:28.014775 22014 log.cc:1196] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/4c52bc43039145e98fff473e73c229aa
I20260923 20:19:28.015457 22014 ts_tablet_manager.cc:1939] T 4c52bc43039145e98fff473e73c229aa P bd49ffff1b4b41beb52ee5ec19a552e7: Deleting consensus metadata
I20260923 20:19:28.017933 21510 catalog_manager.cc:4841] TS bd49ffff1b4b41beb52ee5ec19a552e7 (127.5.83.67:34787): tablet 4c52bc43039145e98fff473e73c229aa (table test-table [id=2d7c1f12c8994269abeff3e05d1ec8ce]) successfully deleted
I20260923 20:19:28.095757 22012 ts_tablet_manager.cc:1939] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: Deleting consensus metadata
I20260923 20:19:28.095794 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 21562
I20260923 20:19:28.115239 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 21714
W20260923 20:19:28.116485 21510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.65:43323 (error 108)
I20260923 20:19:28.119004 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:28.118419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43323 (local address 127.0.0.1:43986)
0923 20:19:28.118669 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:28.118802 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260923 20:19:28.119343 21510 catalog_manager.cc:4584] TS 6448f7afb688403d9ac2c8f667355499 (127.5.83.65:43323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c52bc43039145e98fff473e73c229aa: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43323: connect: Connection refused (error 111)
I20260923 20:19:28.137071 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:28.117599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.5.83.66:42385)
0923 20:19:28.117797 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:28.117804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:28.117826 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:19:28.117893 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:28.117903 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:28.117970 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:28.118098 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:28.120080 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:28.120976 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:28.122437 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:28.122894 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:28.123025 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:28.129779 (+ 6754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:28.129799 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:28.129814 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:28.129851 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:28.133105 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:28.133902 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:28.133910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:28.133912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:28.134051 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:28.134439 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:28.134446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:28.134449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:28.134824 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:28.136217 (+ 1393us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:28.136389 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 20:19:28.136654 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:19:28.138104 21837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:19:28.173128 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:28.172661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43323 (local address 127.0.0.1:43994)
0923 20:19:28.172867 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:28.172985 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 20:19:28.174214 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:28.173860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43323 (local address 127.0.0.1:44010)
0923 20:19:28.174011 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:28.174091 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 20:19:28.234699 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:28.234237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43323 (local address 127.0.0.1:44018)
0923 20:19:28.234448 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:28.234545 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:19:28.235780 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:28.235426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43323 (local address 127.0.0.1:44024)
0923 20:19:28.235580 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:28.235660 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 20:19:28.291301 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 21714
I20260923 20:19:28.309577 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 21845
I20260923 20:19:28.316325 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:28.315721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43323 (local address 127.0.0.1:44036)
0923 20:19:28.316013 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:28.316116 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260923 20:19:28.317569 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:28.317111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43323 (local address 127.0.0.1:44042)
0923 20:19:28.317323 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:28.317424 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260923 20:19:28.331460 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:28.310417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.5.83.67:34787)
0923 20:19:28.310637 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 20:19:28.310645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:19:28.310673 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:19:28.310823 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:19:28.310829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:19:28.310908 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:19:28.311016 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:19:28.312252 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:28.313509 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:28.314857 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:19:28.315559 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:19:28.315690 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:19:28.323574 (+ 7884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:19:28.323591 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:19:28.323613 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:19:28.323678 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:19:28.328036 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:28.329040 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:28.329050 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:28.329053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:28.329252 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:19:28.329617 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:19:28.329622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:19:28.329626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:19:28.330128 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:19:28.330478 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0923 20:19:28.330786 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 20:19:28.331033 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 20:19:28.332296 21968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:19:28.457782 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 21845
I20260923 20:19:28.475039 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 21492
I20260923 20:19:28.476152 21555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:19:28.590679 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:28.590213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:43323 (local address 127.0.0.1:43804)
0923 20:19:28.590412 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:28.590500 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:43323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 20:19:28.590876 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 21492
I20260923 20:19:48.611518 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:46799
--webserver_interface=127.5.83.126
--webserver_port=42513
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:46799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:19:48.956678 22020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:19:48.957047 22020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:19:48.957211 22020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:19:48.964612 22020 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 20:19:48.964776 22020 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 20:19:48.964875 22020 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 20:19:48.964954 22020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:19:48.965036 22020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:19:48.965137 22020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:19:48.965229 22020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:19:48.965308 22020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:19:48.975538 22020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:46799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:46799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=42513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:19:48.977005 22020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:19:48.978335 22020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:19:48.990748 22026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:48.990782 22025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:48.990674 22028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:19:48.992015 22020 server_base.cc:1034] running on GCE node
I20260923 20:19:48.992966 22020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:19:48.994489 22020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:19:48.995749 22020 hybrid_clock.cc:648] HybridClock initialized: now 1790194788995684 us; error 35 us; skew 500 ppm
I20260923 20:19:48.996121 22020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:19:48 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194788, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:19:48.999536 22020 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:19:49.001133 22020 webserver.cc:460] Webserver started at http://127.5.83.126:42513/ using document root <none> and password file <none>
I20260923 20:19:49.001874 22020 fs_manager.cc:359] Metadata directory not provided
I20260923 20:19:49.002010 22020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:19:49.007885 22020 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260923 20:19:49.011065 22035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:49.012683 22020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:19:49.012917 22020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "e277bfca93634e4fbd79f266c2193cb7"
format_stamp: "Formatted at 2026-09-23 20:19:03 on dist-test-slave-vl24"
server_key: "4b2ed2c783707f3e3092ff1f66466f7c"
server_key_iv: "182b616e179a9d36483cc0881c51da7c"
server_key_version: "encryptionkey@0"
I20260923 20:19:49.014035 22020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:19:49.032147 22020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:19:49.036856 22020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:19:49.037184 22020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:19:49.057586 22020 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:46799
I20260923 20:19:49.057646 22086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:46799 every 8 connection(s)
I20260923 20:19:49.059192 22020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:19:49.066533 22087 sys_catalog.cc:263] Verifying existing consensus state
I20260923 20:19:49.069430 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 22020
I20260923 20:19:49.070530 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:43323
--local_ip_for_outbound_sockets=127.5.83.65
--tserver_master_addrs=127.5.83.126:46799
--webserver_port=42753
--webserver_interface=127.5.83.65
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:19:49.070564 22087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Bootstrap starting.
I20260923 20:19:49.095561 22087 log.cc:824] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Log is configured to *not* fsync() on all Append() calls
I20260923 20:19:49.111881 22087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 20:19:49.112620 22087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Bootstrap complete.
I20260923 20:19:49.122494 22087 raft_consensus.cc:348] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } }
I20260923 20:19:49.123478 22087 raft_consensus.cc:729] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e277bfca93634e4fbd79f266c2193cb7, State: Initialized, Role: FOLLOWER
I20260923 20:19:49.124145 22087 consensus_queue.cc:260] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } }
I20260923 20:19:49.124387 22087 raft_consensus.cc:388] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:19:49.124505 22087 raft_consensus.cc:482] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:19:49.124723 22087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 2 FOLLOWER]: Advancing to term 3
I20260923 20:19:49.128700 22087 raft_consensus.cc:504] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } }
I20260923 20:19:49.129241 22087 leader_election.cc:302] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e277bfca93634e4fbd79f266c2193cb7; no voters:
I20260923 20:19:49.129968 22087 leader_election.cc:288] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 20:19:49.130113 22091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 20:19:49.131347 22091 raft_consensus.cc:686] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [term 3 LEADER]: Becoming Leader. State: Replica: e277bfca93634e4fbd79f266c2193cb7, State: Running, Role: LEADER
I20260923 20:19:49.131991 22091 consensus_queue.cc:237] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } }
I20260923 20:19:49.132306 22087 sys_catalog.cc:564] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:19:49.134796 22092 sys_catalog.cc:455] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e277bfca93634e4fbd79f266c2193cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } } }
I20260923 20:19:49.134845 22093 sys_catalog.cc:455] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e277bfca93634e4fbd79f266c2193cb7. Latest consensus state: current_term: 3 leader_uuid: "e277bfca93634e4fbd79f266c2193cb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e277bfca93634e4fbd79f266c2193cb7" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46799 } } }
I20260923 20:19:49.135188 22092 sys_catalog.cc:458] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: This master's current role is: LEADER
I20260923 20:19:49.135238 22093 sys_catalog.cc:458] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7 [sys.catalog]: This master's current role is: LEADER
I20260923 20:19:49.136158 22097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:19:49.148159 22097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:19:49.149750 22097 catalog_manager.cc:1259] Loaded cluster ID: aa81963a24b94261a8e6fea57ff2f34a
I20260923 20:19:49.149930 22097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:19:49.154014 22097 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:19:49.156001 22097 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Loaded TSK: 0
I20260923 20:19:49.156234 22097 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e277bfca93634e4fbd79f266c2193cb7: Loaded TSK: 1
F20260923 20:19:49.157548 22097 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd5debe99ad google::LogMessage::Fail() at ??:0
@ 0x7fd5debec88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd5debe94de google::LogMessage::Flush() at ??:0
@ 0x7fd5debeaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd5e294d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd5e293c73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd5f46181c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd5f4617783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd5f46a32a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fd5f46a30c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fd5f46ed5e9 std::function<>::operator()() at ??:0
@ 0x7fd5f461bcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd5f4619c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd5f46a1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd5f46a1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd5e1835345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd5e183891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd5e1838799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd5e1813eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd5e4aed6db start_thread at ??:0
@ 0x7fd5dda8361f clone at ??:0
W20260923 20:19:49.481446 22089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:19:49.481807 22089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:19:49.481981 22089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:19:49.489656 22089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:19:49.489836 22089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:19:49.490015 22089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:19:49.500263 22089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:43323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=42753
--tserver_master_addrs=127.5.83.126:46799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:19:49.501730 22089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:19:49.503116 22089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:19:49.516703 22134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:49.516757 22137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:49.516675 22135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:19:49.517477 22089 server_base.cc:1034] running on GCE node
I20260923 20:19:49.518114 22089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:19:49.519094 22089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:19:49.520306 22089 hybrid_clock.cc:648] HybridClock initialized: now 1790194789520241 us; error 60 us; skew 500 ppm
I20260923 20:19:49.520689 22089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:19:49 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194789, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:19:49.523891 22089 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:19:49.525496 22089 webserver.cc:460] Webserver started at http://127.5.83.65:42753/ using document root <none> and password file <none>
I20260923 20:19:49.526291 22089 fs_manager.cc:359] Metadata directory not provided
I20260923 20:19:49.526446 22089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:19:49.532178 22089 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 20:19:49.535169 22144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:49.536685 22089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:19:49.536870 22089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "6448f7afb688403d9ac2c8f667355499"
format_stamp: "Formatted at 2026-09-23 20:19:03 on dist-test-slave-vl24"
server_key: "424d1305d2e83e904fa29f1d1c1b8522"
server_key_iv: "b2b438ffc6a3d23404b3ffea1f22fb67"
server_key_version: "encryptionkey@0"
I20260923 20:19:49.537860 22089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:19:49.561517 22089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:19:49.565699 22089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:19:49.566008 22089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:19:49.567071 22089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:19:49.568938 22151 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260923 20:19:49.574301 22151 ts_tablet_manager.cc:1852] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260923 20:19:49.575330 22151 ts_tablet_manager.cc:1905] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:19:49.582962 22151 ts_tablet_manager.cc:1918] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:19:49.583266 22151 ts_tablet_manager.cc:1939] T 4c52bc43039145e98fff473e73c229aa P 6448f7afb688403d9ac2c8f667355499: Deleting consensus metadata
I20260923 20:19:49.583840 22089 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260923 20:19:49.584093 22089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.000s
I20260923 20:19:49.584229 22089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:19:49.584324 22089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:49.619835 22089 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:43323
I20260923 20:19:49.619962 22257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:43323 every 8 connection(s)
I20260923 20:19:49.621235 22089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:19:49.626343 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:49.624401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:37121)
0923 20:19:49.625258 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:49.625482 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":430,"threads_started":1}
W20260923 20:19:49.627049 22258 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:46799 (0 consecutive failures): Network error: Failed to ping master at 127.5.83.126:46799: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
I20260923 20:19:49.628014 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:49.627624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:46389)
0923 20:19:49.627826 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:49.627895 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 20:19:49.628986 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:49.628628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:54049)
0923 20:19:49.628762 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:49.628863 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 20:19:49.629232 22258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:19:49.631834 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 22089
I20260923 20:19:49.632516 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:42385
--local_ip_for_outbound_sockets=127.5.83.66
--tserver_master_addrs=127.5.83.126:46799
--webserver_port=40207
--webserver_interface=127.5.83.66
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:19:49.977007 22261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:19:49.977324 22261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:19:49.977427 22261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:19:49.985352 22261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:19:49.985482 22261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:19:49.985607 22261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:19:49.996132 22261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:42385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=40207
--tserver_master_addrs=127.5.83.126:46799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:19:49.997602 22261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:19:49.999042 22261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:19:50.013105 22266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:50.013161 22269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:50.013070 22267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:19:50.014394 22261 server_base.cc:1034] running on GCE node
I20260923 20:19:50.015157 22261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:19:50.016215 22261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:19:50.017477 22261 hybrid_clock.cc:648] HybridClock initialized: now 1790194790017356 us; error 106 us; skew 500 ppm
I20260923 20:19:50.017828 22261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:19:50 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194790, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:19:50.021389 22261 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:19:50.022928 22261 webserver.cc:460] Webserver started at http://127.5.83.66:40207/ using document root <none> and password file <none>
I20260923 20:19:50.023624 22261 fs_manager.cc:359] Metadata directory not provided
I20260923 20:19:50.023741 22261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:19:50.029531 22261 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 20:19:50.032405 22276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:50.033855 22261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:19:50.034010 22261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "c0bfd44eba284296873dbd9465c781f3"
format_stamp: "Formatted at 2026-09-23 20:19:04 on dist-test-slave-vl24"
server_key: "67d5abec5c07b0ed8cfdd9abb97b5e44"
server_key_iv: "b20c2140cc78cd468e0f964c7bd7fdc2"
server_key_version: "encryptionkey@0"
I20260923 20:19:50.035037 22261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:19:50.061239 22261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:19:50.066641 22261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:19:50.066977 22261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:19:50.068074 22261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:19:50.069624 22261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:19:50.069792 22261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:50.069936 22261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:19:50.070039 22261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:50.104420 22261 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:42385
I20260923 20:19:50.104620 22388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:42385 every 8 connection(s)
I20260923 20:19:50.105891 22261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:19:50.110755 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 22261
I20260923 20:19:50.111078 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:50.109057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:43611)
0923 20:19:50.109838 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:50.110098 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":305,"threads_started":1}
I20260923 20:19:50.111523 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:34787
--local_ip_for_outbound_sockets=127.5.83.67
--tserver_master_addrs=127.5.83.126:46799
--webserver_port=38517
--webserver_interface=127.5.83.67
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:19:50.112071 22389 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:46799 (0 consecutive failures): Network error: Failed to ping master at 127.5.83.126:46799: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
I20260923 20:19:50.113165 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:50.112750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:51073)
0923 20:19:50.112975 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:50.113049 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260923 20:19:50.114252 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:50.113924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:36899)
0923 20:19:50.114067 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:50.114142 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 20:19:50.114557 22389 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 20:19:50.447464 22392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:19:50.447743 22392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:19:50.447839 22392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:19:50.455008 22392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:19:50.455116 22392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:19:50.455245 22392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:19:50.465061 22392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:34787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=38517
--tserver_master_addrs=127.5.83.126:46799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:19:50.466360 22392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:19:50.467635 22392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:19:50.480917 22397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:50.480808 22398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:19:50.480810 22400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:19:50.481875 22392 server_base.cc:1034] running on GCE node
I20260923 20:19:50.482523 22392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:19:50.483479 22392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:19:50.484697 22392 hybrid_clock.cc:648] HybridClock initialized: now 1790194790484667 us; error 55 us; skew 500 ppm
I20260923 20:19:50.485033 22392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:19:50 dist-test-slave-vl24 krb5kdc[20941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194790, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:19:50.493345 22392 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:19:50.494972 22392 webserver.cc:460] Webserver started at http://127.5.83.67:38517/ using document root <none> and password file <none>
I20260923 20:19:50.495663 22392 fs_manager.cc:359] Metadata directory not provided
I20260923 20:19:50.495762 22392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:19:50.501139 22392 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:19:50.503698 22407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:50.505067 22392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:19:50.505196 22392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "bd49ffff1b4b41beb52ee5ec19a552e7"
format_stamp: "Formatted at 2026-09-23 20:19:04 on dist-test-slave-vl24"
server_key: "92453e8fe57b539b0859c0958314d55a"
server_key_iv: "2ca4da513c0e7e0255c7e4d382555fcb"
server_key_version: "encryptionkey@0"
I20260923 20:19:50.506119 22392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:19:50.529867 22392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:19:50.534665 22392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:19:50.534907 22392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:19:50.535843 22392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:19:50.537266 22392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:19:50.537355 22392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:50.537436 22392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:19:50.537489 22392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:19:50.571028 22392 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:34787
I20260923 20:19:50.571125 22519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:34787 every 8 connection(s)
I20260923 20:19:50.572446 22392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:19:50.577133 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:50.575314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:42305)
0923 20:19:50.576005 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:50.576225 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":271,"threads_started":1}
W20260923 20:19:50.577812 22520 heartbeater.cc:644] Failed to heartbeat to 127.5.83.126:46799 (0 consecutive failures): Network error: Failed to ping master at 127.5.83.126:46799: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
I20260923 20:19:50.578620 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 22392
I20260923 20:19:50.578646 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:50.578309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:47173)
0923 20:19:50.578477 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:50.578548 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:19:50.579573 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:50.579248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:56509)
0923 20:19:50.579396 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:50.579470 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 20:19:50.579929 22520 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 20:19:50.630529 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:50.629780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:54745)
0923 20:19:50.630229 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:50.630308 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":223,"threads_started":1}
I20260923 20:19:51.116029 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:51.115187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:58213)
0923 20:19:51.115679 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:51.115796 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":238,"threads_started":1}
I20260923 20:19:51.581503 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:51.580668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:38009)
0923 20:19:51.581162 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:51.581260 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":259,"threads_started":1}
I20260923 20:19:51.632333 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:51.631572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:56919)
0923 20:19:51.632064 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:51.632155 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":275,"threads_started":1}
I20260923 20:19:52.117877 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:52.117082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:48407)
0923 20:19:52.117598 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:52.117685 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":264,"threads_started":1}
I20260923 20:19:52.583194 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:52.582455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:56715)
0923 20:19:52.582942 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:52.583029 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":245,"threads_started":1}
I20260923 20:19:52.634140 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:52.633409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:36629)
0923 20:19:52.633883 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:52.633974 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":234,"threads_started":1}
I20260923 20:19:53.119493 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:53.118765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:51643)
0923 20:19:53.119213 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:53.119298 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260923 20:19:53.584916 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:53.584136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:49229)
0923 20:19:53.584650 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:53.584733 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":248,"threads_started":1}
I20260923 20:19:53.636087 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:53.635242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:43887)
0923 20:19:53.635771 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:53.635870 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":276,"threads_started":1}
I20260923 20:19:54.121258 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:54.120492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:52167)
0923 20:19:54.120992 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:54.121075 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":232,"threads_started":1}
I20260923 20:19:54.586503 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:54.585792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:55429)
0923 20:19:54.586237 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:54.586321 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":221,"threads_started":1}
I20260923 20:19:54.637888 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:54.637208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:35797)
0923 20:19:54.637655 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:54.637730 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":223,"threads_started":1}
I20260923 20:19:55.123060 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:55.122269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:44657)
0923 20:19:55.122778 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:55.122883 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":277,"threads_started":1}
I20260923 20:19:55.588279 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:55.587530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:51773)
0923 20:19:55.588021 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:55.588105 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
I20260923 20:19:55.639657 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:55.638869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:42373)
0923 20:19:55.639396 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:55.639478 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":267,"threads_started":1}
I20260923 20:19:56.125000 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:56.124077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:51059)
0923 20:19:56.124696 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:56.124821 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":286,"threads_started":1}
I20260923 20:19:56.589980 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:56.589211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:46595)
0923 20:19:56.589729 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:56.589818 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":295,"threads_started":1}
I20260923 20:19:56.641543 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:56.640675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:33501)
0923 20:19:56.641240 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:56.641329 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":260,"threads_started":1}
I20260923 20:19:57.126931 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:57.126121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:39913)
0923 20:19:57.126637 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:57.126722 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":212,"threads_started":1}
I20260923 20:19:57.591729 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:57.590910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:41533)
0923 20:19:57.591427 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:57.591520 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":242,"threads_started":1}
I20260923 20:19:57.643219 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:57.642450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:49193)
0923 20:19:57.642952 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:57.643038 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":269,"threads_started":1}
I20260923 20:19:58.128681 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:58.127888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:54643)
0923 20:19:58.128349 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:58.128436 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":243,"threads_started":1}
I20260923 20:19:58.593608 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:58.592794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:57813)
0923 20:19:58.593343 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:58.593435 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":303,"threads_started":1}
I20260923 20:19:58.644904 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:58.644117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:47621)
0923 20:19:58.644657 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:58.644749 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":235,"threads_started":1}
I20260923 20:19:59.130398 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:59.129659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:41787)
0923 20:19:59.130143 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:59.130224 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":247,"threads_started":1}
I20260923 20:19:59.595273 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:59.594506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:50013)
0923 20:19:59.595001 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:59.595087 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":238,"threads_started":1}
I20260923 20:19:59.646777 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:19:59.645976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:40233)
0923 20:19:59.646523 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 20:19:59.646601 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":315,"threads_started":1}
I20260923 20:20:00.132086 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:00.131359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:57275)
0923 20:20:00.131839 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:00.131926 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":263,"threads_started":1}
I20260923 20:20:00.597090 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:00.596269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:52389)
0923 20:20:00.596824 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:00.596922 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":331,"threads_started":1}
I20260923 20:20:00.648716 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:00.647780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:45377)
0923 20:20:00.648351 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:00.648474 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":307,"threads_started":1}
I20260923 20:20:01.133888 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:01.133127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:43249)
0923 20:20:01.133589 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:01.133687 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":240,"threads_started":1}
I20260923 20:20:01.598825 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:01.598024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:55013)
0923 20:20:01.598547 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:01.598647 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":298,"threads_started":1}
I20260923 20:20:01.650436 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:01.649688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:60399)
0923 20:20:01.650172 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:01.650260 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":247,"threads_started":1}
I20260923 20:20:02.135576 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:02.134865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:52307)
0923 20:20:02.135322 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:02.135408 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":226,"threads_started":1}
I20260923 20:20:02.600775 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:02.599848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:47797)
0923 20:20:02.600420 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:02.600558 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":237,"threads_started":1}
I20260923 20:20:02.652330 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:02.651522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:37693)
0923 20:20:02.652036 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:02.652149 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":248,"threads_started":1}
I20260923 20:20:03.137678 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:03.136744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:48881)
0923 20:20:03.137317 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:03.137416 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":308,"threads_started":1}
I20260923 20:20:03.602564 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:03.601795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:51243)
0923 20:20:03.602276 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:03.602376 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":241,"threads_started":1}
I20260923 20:20:03.653973 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:03.653259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:57309)
0923 20:20:03.653699 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:03.653778 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":229,"threads_started":1}
I20260923 20:20:04.139503 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:04.138694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:32831)
0923 20:20:04.139199 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:04.139315 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":268,"threads_started":1}
I20260923 20:20:04.604291 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:04.603542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:50025)
0923 20:20:04.604018 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:04.604107 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":253,"threads_started":1}
I20260923 20:20:04.655812 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:04.655050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.65:36713)
0923 20:20:04.655532 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:04.655644 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":232,"threads_started":1}
I20260923 20:20:05.141574 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:05.140717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.66:34047)
0923 20:20:05.141264 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:05.141385 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":296,"threads_started":1}
I20260923 20:20:05.581743 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 22089
I20260923 20:20:05.600126 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:05.582976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.5.83.65:43323)
0923 20:20:05.583617 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:05.583640 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:05.583994 (+ 354us) server_negotiation.cc:407] Connection header received
0923 20:20:05.585897 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:05.585909 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:05.586329 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:05.586624 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:05.587952 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:05.589288 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:05.590479 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:05.591110 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:05.591246 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:05.593871 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:05.593879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:05.593894 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:05.593933 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:05.597043 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:05.597710 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:05.597716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:05.597722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:05.597879 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:05.598245 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:05.598248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:05.598251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:05.598782 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:05.599007 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:05.599370 (+ 363us) server_negotiation.cc:299] Negotiation successful
0923 20:20:05.599821 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":278,"threads_started":1}
I20260923 20:20:05.601637 22253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:05.606473 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:05.605585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46799 (local address 127.5.83.67:32917)
0923 20:20:05.606076 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:05.606237 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.126:46799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":251,"threads_started":1}
I20260923 20:20:05.694754 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 22089
I20260923 20:20:05.714537 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 22261
I20260923 20:20:05.734073 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:05.715552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.5.83.66:42385)
0923 20:20:05.716094 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:05.716109 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:05.716145 (+ 36us) server_negotiation.cc:407] Connection header received
0923 20:20:05.717859 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:05.717869 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:05.718321 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:05.718719 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:05.720076 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:05.721365 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:05.722694 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:05.723420 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:05.723577 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:05.726593 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:05.726603 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:05.726621 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:05.726676 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:05.729949 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:05.730774 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:05.730782 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:05.730790 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:05.731012 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:05.731454 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:05.731458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:05.731460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:05.732046 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:05.732306 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:05.733269 (+ 963us) server_negotiation.cc:299] Negotiation successful
0923 20:20:05.733745 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":244,"threads_started":1}
I20260923 20:20:05.735569 22384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:05.824718 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 22261
I20260923 20:20:05.845877 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 22392
I20260923 20:20:05.865841 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:05.846942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.5.83.67:34787)
0923 20:20:05.847580 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:05.847595 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:05.847640 (+ 45us) server_negotiation.cc:407] Connection header received
0923 20:20:05.849395 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:05.849409 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:05.849857 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:05.850172 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:05.851657 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:05.853137 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:05.854441 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:05.855174 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:05.855318 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:05.858176 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:05.858184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:05.858201 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:05.858247 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:05.861938 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:05.862741 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:05.862748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:05.862756 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:05.862978 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:05.863441 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:05.863445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:05.863447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:05.864058 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:05.864316 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:05.864971 (+ 655us) server_negotiation.cc:299] Negotiation successful
0923 20:20:05.865357 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":304,"threads_started":1}
I20260923 20:20:05.867432 22515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:05.953375 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 22392
2026-09-23T20:20:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64621 ms)
[----------] 30 tests from SecurityITest (299453 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:06 dist-test-slave-vl24 krb5kdc[22578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:06 dist-test-slave-vl24 krb5kdc[22578](info): set up 2 sockets
Sep 23 20:20:06 dist-test-slave-vl24 krb5kdc[22578](info): commencing operation
krb5kdc: starting...
W20260923 20:20:07.069787 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:07 dist-test-slave-vl24 krb5kdc[22578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:20:07.340178 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36425
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:36425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:07.683207 22591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:07.683565 22591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:07.683744 22591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:07.691006 22591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:07.691179 22591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:07.691291 22591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:07.691402 22591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:07.691475 22591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:07.701534 22591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:36425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:07.702937 22591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:07.704289 22591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:07.717142 22596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:07.717059 22597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:07.717092 22599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:07.722071 22591 server_base.cc:1034] running on GCE node
I20260923 20:20:07.723057 22591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:07.724572 22591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:07.725790 22591 hybrid_clock.cc:648] HybridClock initialized: now 1790194807725747 us; error 35 us; skew 500 ppm
I20260923 20:20:07.726151 22591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:07 dist-test-slave-vl24 krb5kdc[22578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194807, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:07.733510 22591 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:20:07.735093 22591 webserver.cc:460] Webserver started at http://127.5.83.126:34915/ using document root <none> and password file <none>
I20260923 20:20:07.735778 22591 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:07.735879 22591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:07.736135 22591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:07.739146 22591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "780f79d1de9c4a87aa139bfdb750d738"
format_stamp: "Formatted at 2026-09-23 20:20:07 on dist-test-slave-vl24"
server_key: "ddb599be9969dcd766312e46fe40125d"
server_key_iv: "89d630a8f5fcb6e97a32312f8706f761"
server_key_version: "encryptionkey@0"
I20260923 20:20:07.739852 22591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "780f79d1de9c4a87aa139bfdb750d738"
format_stamp: "Formatted at 2026-09-23 20:20:07 on dist-test-slave-vl24"
server_key: "ddb599be9969dcd766312e46fe40125d"
server_key_iv: "89d630a8f5fcb6e97a32312f8706f761"
server_key_version: "encryptionkey@0"
I20260923 20:20:07.745085 22591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:20:07.748649 22606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:07.750036 22591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:20:07.750224 22591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "780f79d1de9c4a87aa139bfdb750d738"
format_stamp: "Formatted at 2026-09-23 20:20:07 on dist-test-slave-vl24"
server_key: "ddb599be9969dcd766312e46fe40125d"
server_key_iv: "89d630a8f5fcb6e97a32312f8706f761"
server_key_version: "encryptionkey@0"
I20260923 20:20:07.750481 22591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:07.776496 22591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:07.781275 22591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:07.781605 22591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:07.801362 22591 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:36425
I20260923 20:20:07.801384 22657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:36425 every 8 connection(s)
I20260923 20:20:07.802757 22591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:07.806476 22658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:07.806556 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 22591
I20260923 20:20:07.806783 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:07.807483 5453 external_mini_cluster.cc:1428] Setting key f79fb394b343f6fd4c1b046cd46a3877
I20260923 20:20:07.821257 22658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738: Bootstrap starting.
Sep 23 20:20:07 dist-test-slave-vl24 krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:07.825519 22658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:07.826674 22658 log.cc:824] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:07.830358 22658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738: No bootstrap required, opened a new log
I20260923 20:20:07.835089 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:07.809628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.5.83.126:36425)
0923 20:20:07.810543 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:07.810559 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:07.810614 (+ 55us) server_negotiation.cc:407] Connection header received
0923 20:20:07.813414 (+ 2800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:07.813453 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:07.814001 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:07.814476 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:07.815880 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:07.817935 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:07.819376 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:07.820413 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:07.820623 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:07.825913 (+ 5290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:07.825922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:07.825936 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:07.825979 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:07.829998 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:07.830861 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:07.830875 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:07.830887 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:07.831110 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:07.831562 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:07.831567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:07.831570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:07.832296 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:07.832665 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:07.833090 (+ 425us) server_negotiation.cc:299] Negotiation successful
0923 20:20:07.833603 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":378,"threads_started":1}
I20260923 20:20:07.840020 22658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "780f79d1de9c4a87aa139bfdb750d738" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36425 } }
I20260923 20:20:07.840483 22658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:07.840629 22658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 780f79d1de9c4a87aa139bfdb750d738, State: Initialized, Role: FOLLOWER
I20260923 20:20:07.841145 22658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "780f79d1de9c4a87aa139bfdb750d738" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36425 } }
I20260923 20:20:07.841351 22658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:07.841447 22658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:07.841595 22658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:07.844128 22658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "780f79d1de9c4a87aa139bfdb750d738" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36425 } }
I20260923 20:20:07.844648 22658 leader_election.cc:302] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 780f79d1de9c4a87aa139bfdb750d738; no voters:
I20260923 20:20:07.845192 22658 leader_election.cc:288] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:07.845331 22663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:07.846120 22663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [term 1 LEADER]: Becoming Leader. State: Replica: 780f79d1de9c4a87aa139bfdb750d738, State: Running, Role: LEADER
I20260923 20:20:07.846633 22663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "780f79d1de9c4a87aa139bfdb750d738" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36425 } }
I20260923 20:20:07.847482 22658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:07.849577 22665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 780f79d1de9c4a87aa139bfdb750d738. Latest consensus state: current_term: 1 leader_uuid: "780f79d1de9c4a87aa139bfdb750d738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "780f79d1de9c4a87aa139bfdb750d738" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36425 } } }
I20260923 20:20:07.850024 22665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:07.850852 22664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "780f79d1de9c4a87aa139bfdb750d738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "780f79d1de9c4a87aa139bfdb750d738" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36425 } } }
I20260923 20:20:07.851231 22664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:07.855134 22671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:07.859849 22671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:07.868331 22671 catalog_manager.cc:1347] Generated new cluster ID: a28ea340cdf8468e97e7ee7251eca752
I20260923 20:20:07.868462 22671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:07.878036 22671 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:07.879369 22671 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:07.887493 22671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738: Generated new TSK 0
I20260923 20:20:07.888468 22671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:20:07.956676 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36425
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:08.296872 22684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:08.297144 22684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:08.297238 22684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:08.304368 22684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:08.304484 22684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:08.304682 22684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:20:08.314766 22684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:08.316054 22684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:08.317426 22684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:08.331095 22692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:08.331176 22690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:08.331207 22689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:08.332363 22684 server_base.cc:1034] running on GCE node
I20260923 20:20:08.333096 22684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:08.334311 22684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:08.335547 22684 hybrid_clock.cc:648] HybridClock initialized: now 1790194808335495 us; error 43 us; skew 500 ppm
I20260923 20:20:08.335898 22684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:08 dist-test-slave-vl24 krb5kdc[22578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194808, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:08.339200 22684 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:20:08.340749 22684 webserver.cc:460] Webserver started at http://127.5.83.65:40687/ using document root <none> and password file <none>
I20260923 20:20:08.341421 22684 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:08.341526 22684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:08.341789 22684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:08.344726 22684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "ae0196003bce48bf8d03a7ef5b0dfb10"
format_stamp: "Formatted at 2026-09-23 20:20:08 on dist-test-slave-vl24"
server_key: "4409c0c691d4e363b0845b24a0ac36e2"
server_key_iv: "a9b7eaa5b90522a106ac0474eddc85d8"
server_key_version: "encryptionkey@0"
I20260923 20:20:08.345475 22684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "ae0196003bce48bf8d03a7ef5b0dfb10"
format_stamp: "Formatted at 2026-09-23 20:20:08 on dist-test-slave-vl24"
server_key: "4409c0c691d4e363b0845b24a0ac36e2"
server_key_iv: "a9b7eaa5b90522a106ac0474eddc85d8"
server_key_version: "encryptionkey@0"
I20260923 20:20:08.350677 22684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:20:08.354264 22699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:08.355801 22684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:20:08.355981 22684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "ae0196003bce48bf8d03a7ef5b0dfb10"
format_stamp: "Formatted at 2026-09-23 20:20:08 on dist-test-slave-vl24"
server_key: "4409c0c691d4e363b0845b24a0ac36e2"
server_key_iv: "a9b7eaa5b90522a106ac0474eddc85d8"
server_key_version: "encryptionkey@0"
I20260923 20:20:08.356175 22684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:08.378001 22684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:08.382619 22684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:08.382957 22684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:08.383936 22684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:08.385607 22684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:08.385696 22684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:08.385782 22684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:08.385828 22684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:08.420804 22684 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:34199
I20260923 20:20:08.420878 22811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:34199 every 8 connection(s)
I20260923 20:20:08.422129 22684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:20:08.423969 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 22684
I20260923 20:20:08.424188 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:20:08.425120 5453 external_mini_cluster.cc:1428] Setting key 6e23eaecbbfec9499aae710e8a861cc8
Sep 23 20:20:08 dist-test-slave-vl24 krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194808, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:08.443730 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:08.424779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:46913 (local address 127.5.83.126:36425)
0923 20:20:08.425325 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:08.425331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:08.426310 (+ 979us) server_negotiation.cc:407] Connection header received
0923 20:20:08.428180 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:08.428188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:08.428290 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:08.428411 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:08.429884 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:08.431020 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:08.432558 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:08.433289 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:08.433440 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:08.437226 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:08.437231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:08.437234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:08.437266 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:08.440249 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:08.441321 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:08.441325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:08.441327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:08.441465 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:08.441964 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:08.441967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:08.441969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:08.442229 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:08.442338 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:08.443278 (+ 940us) server_negotiation.cc:299] Negotiation successful
0923 20:20:08.443466 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":281,"threads_started":1}
I20260923 20:20:08.445384 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:08.425324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36425 (local address 127.5.83.65:46913)
0923 20:20:08.426146 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:08.426199 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:08.427908 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:08.428667 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:08.428675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:08.429172 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:08.429697 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:08.429713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:08.431172 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:08.431176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:08.432378 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:08.432389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:08.433499 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:08.433503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:08.433699 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:08.434403 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:20:08.434424 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:08.437002 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:08.440418 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:08.440430 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:08.440453 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:08.441188 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:08.441580 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:08.441587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:08.441593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:08.441853 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:08.442364 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:08.442375 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:08.442860 (+ 485us) client_negotiation.cc:769] Sending connection context
0923 20:20:08.443183 (+ 323us) client_negotiation.cc:240] Negotiation successful
0923 20:20:08.443781 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":331,"threads_started":1}
I20260923 20:20:08.446972 22812 heartbeater.cc:347] Connected to a master server at 127.5.83.126:36425
I20260923 20:20:08.447314 22812 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:08.448067 22812 heartbeater.cc:511] Master 127.5.83.126:36425 requested a full tablet report, sending...
I20260923 20:20:08.450219 22623 ts_manager.cc:194] Registered new tserver with Master: ae0196003bce48bf8d03a7ef5b0dfb10 (127.5.83.65:34199)
I20260923 20:20:08.451659 22623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:46913
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:20:08.488411 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36425
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:08.829515 22818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:08.829804 22818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:08.829898 22818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:08.837394 22818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:08.837502 22818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:08.837615 22818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:20:08.847304 22818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:08.848739 22818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:08.850069 22818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:08.863341 22823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:08.863377 22824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:08.863461 22826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:08.864240 22818 server_base.cc:1034] running on GCE node
I20260923 20:20:08.864892 22818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:08.865821 22818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:08.867012 22818 hybrid_clock.cc:648] HybridClock initialized: now 1790194808866966 us; error 38 us; skew 500 ppm
I20260923 20:20:08.867326 22818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:08 dist-test-slave-vl24 krb5kdc[22578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194808, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:08.870489 22818 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:20:08.871949 22818 webserver.cc:460] Webserver started at http://127.5.83.66:33691/ using document root <none> and password file <none>
I20260923 20:20:08.872716 22818 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:08.872849 22818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:08.873169 22818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:08.876011 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "2ef52514966c4a2595746055ea76512c"
format_stamp: "Formatted at 2026-09-23 20:20:08 on dist-test-slave-vl24"
server_key: "e77a248717190b7c1ef382d25a84c194"
server_key_iv: "2433da6ba6c7626167d7ab79a71cb072"
server_key_version: "encryptionkey@0"
I20260923 20:20:08.876876 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "2ef52514966c4a2595746055ea76512c"
format_stamp: "Formatted at 2026-09-23 20:20:08 on dist-test-slave-vl24"
server_key: "e77a248717190b7c1ef382d25a84c194"
server_key_iv: "2433da6ba6c7626167d7ab79a71cb072"
server_key_version: "encryptionkey@0"
I20260923 20:20:08.881861 22818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:20:08.885151 22833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:08.886372 22818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:20:08.886535 22818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "2ef52514966c4a2595746055ea76512c"
format_stamp: "Formatted at 2026-09-23 20:20:08 on dist-test-slave-vl24"
server_key: "e77a248717190b7c1ef382d25a84c194"
server_key_iv: "2433da6ba6c7626167d7ab79a71cb072"
server_key_version: "encryptionkey@0"
I20260923 20:20:08.886729 22818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:08.900468 22818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:08.904475 22818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:08.904809 22818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:08.905776 22818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:08.907125 22818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:08.907195 22818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:08.907356 22818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:08.907418 22818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:08.940773 22818 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:38449
I20260923 20:20:08.940807 22945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:38449 every 8 connection(s)
I20260923 20:20:08.942103 22818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:20:08.945076 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 22818
I20260923 20:20:08.945268 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:20:08.945874 5453 external_mini_cluster.cc:1428] Setting key cd500ead3d33215634d9a8f870aeebbe
Sep 23 20:20:08 dist-test-slave-vl24 krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194808, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:08.964756 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:08.944597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:60865 (local address 127.5.83.126:36425)
0923 20:20:08.945062 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:08.945069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:08.946186 (+ 1117us) server_negotiation.cc:407] Connection header received
0923 20:20:08.948446 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:08.948450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:08.948579 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:08.948666 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:08.950729 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:08.951465 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:08.953895 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:08.954321 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:08.954424 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:08.958587 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:08.958592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:08.958595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:08.958626 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:08.961699 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:08.962416 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:08.962422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:08.962425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:08.962559 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:08.962998 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:08.963003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:08.963007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:08.963289 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:08.963452 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:08.964127 (+ 675us) server_negotiation.cc:299] Negotiation successful
0923 20:20:08.964344 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":234,"threads_started":1}
I20260923 20:20:08.965723 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:08.945152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36425 (local address 127.5.83.66:60865)
0923 20:20:08.946012 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:08.946061 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:08.948144 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:08.948939 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:08.948954 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:08.949737 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:08.950557 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:08.950585 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:08.951623 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:08.951631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:08.953748 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:08.953766 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:08.954540 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:08.954549 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:08.954888 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:08.955651 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:20:08.955687 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:08.958353 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:08.961847 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:08.961856 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:08.961873 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:08.962281 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:08.962668 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:08.962673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:08.962676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:08.962871 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:08.963415 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:08.963423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:08.963760 (+ 337us) client_negotiation.cc:769] Sending connection context
0923 20:20:08.964023 (+ 263us) client_negotiation.cc:240] Negotiation successful
0923 20:20:08.964560 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":322,"threads_started":1}
I20260923 20:20:08.967207 22946 heartbeater.cc:347] Connected to a master server at 127.5.83.126:36425
I20260923 20:20:08.967509 22946 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:08.968205 22946 heartbeater.cc:511] Master 127.5.83.126:36425 requested a full tablet report, sending...
I20260923 20:20:08.969806 22623 ts_manager.cc:194] Registered new tserver with Master: 2ef52514966c4a2595746055ea76512c (127.5.83.66:38449)
I20260923 20:20:08.971131 22623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:60865
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:20:09.008265 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36425
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:09.352952 22952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:09.353235 22952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:09.353420 22952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:09.361516 22952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:09.361634 22952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:09.361841 22952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:20:09.372141 22952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:09.373584 22952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:09.374925 22952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:09.388684 22960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:09.388641 22957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:09.388716 22958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:09.389290 22952 server_base.cc:1034] running on GCE node
I20260923 20:20:09.390033 22952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:09.391345 22952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:09.392639 22952 hybrid_clock.cc:648] HybridClock initialized: now 1790194809392584 us; error 35 us; skew 500 ppm
I20260923 20:20:09.393021 22952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:09 dist-test-slave-vl24 krb5kdc[22578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194809, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:09.396142 22952 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:20:09.397914 22952 webserver.cc:460] Webserver started at http://127.5.83.67:36963/ using document root <none> and password file <none>
I20260923 20:20:09.398663 22952 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:09.398808 22952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:09.399132 22952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:09.402202 22952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "46849de13f0348109cde9dae307215fc"
format_stamp: "Formatted at 2026-09-23 20:20:09 on dist-test-slave-vl24"
server_key: "c5b4e5860f1cbe9783d1b87c9eb0327a"
server_key_iv: "6a2c9e60c3ff0f396b3220d20cd848a3"
server_key_version: "encryptionkey@0"
I20260923 20:20:09.403021 22952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "46849de13f0348109cde9dae307215fc"
format_stamp: "Formatted at 2026-09-23 20:20:09 on dist-test-slave-vl24"
server_key: "c5b4e5860f1cbe9783d1b87c9eb0327a"
server_key_iv: "6a2c9e60c3ff0f396b3220d20cd848a3"
server_key_version: "encryptionkey@0"
I20260923 20:20:09.408257 22952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 20:20:09.411782 22967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:09.413107 22952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:20:09.413286 22952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "46849de13f0348109cde9dae307215fc"
format_stamp: "Formatted at 2026-09-23 20:20:09 on dist-test-slave-vl24"
server_key: "c5b4e5860f1cbe9783d1b87c9eb0327a"
server_key_iv: "6a2c9e60c3ff0f396b3220d20cd848a3"
server_key_version: "encryptionkey@0"
I20260923 20:20:09.413548 22952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:09.429358 22952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:09.433468 22952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:09.433758 22952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:09.434727 22952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:09.436024 22952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:09.436127 22952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:09.436278 22952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:09.436370 22952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:09.454691 22812 heartbeater.cc:503] Master 127.5.83.126:36425 was elected leader, sending a full tablet report...
I20260923 20:20:09.472149 22952 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:45985
I20260923 20:20:09.472241 23079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:45985 every 8 connection(s)
I20260923 20:20:09.473567 22952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:20:09.478962 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 22952
I20260923 20:20:09.479166 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:20:09.479619 5453 external_mini_cluster.cc:1428] Setting key ef9ecfac253694bda9fb9256b49a1850
Sep 23 20:20:09 dist-test-slave-vl24 krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194809, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:09.494094 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.476075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:55977 (local address 127.5.83.126:36425)
0923 20:20:09.476591 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:09.476595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:09.477387 (+ 792us) server_negotiation.cc:407] Connection header received
0923 20:20:09.479293 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:09.479299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:09.479392 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:09.479509 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:09.481435 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.482585 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.484129 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.484516 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.484693 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.488027 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:09.488038 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:09.488041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:09.488074 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:09.490875 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:09.491695 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:09.491702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:09.491706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:09.491845 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:09.492255 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:09.492260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:09.492264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:09.492613 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:09.492801 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:09.493490 (+ 689us) server_negotiation.cc:299] Negotiation successful
0923 20:20:09.493713 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":258,"threads_started":1}
I20260923 20:20:09.495034 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.476446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36425 (local address 127.5.83.67:55977)
0923 20:20:09.477223 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:09.477266 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:09.478984 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:09.479774 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:09.479798 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:09.480460 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:09.481267 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:09.481288 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.482732 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.482738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:09.483977 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:09.483989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.484790 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.484797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:09.485030 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.485700 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:20:09.485723 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:09.487811 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:09.491021 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:09.491031 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:09.491048 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:09.491578 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:09.491953 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:09.491958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:09.491962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:09.492140 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:09.492763 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:09.492772 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:09.493115 (+ 343us) client_negotiation.cc:769] Sending connection context
0923 20:20:09.493390 (+ 275us) client_negotiation.cc:240] Negotiation successful
0923 20:20:09.493923 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":278,"threads_started":1}
I20260923 20:20:09.496316 23080 heartbeater.cc:347] Connected to a master server at 127.5.83.126:36425
I20260923 20:20:09.496677 23080 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:09.497385 23080 heartbeater.cc:511] Master 127.5.83.126:36425 requested a full tablet report, sending...
I20260923 20:20:09.498762 22623 ts_manager.cc:194] Registered new tserver with Master: 46849de13f0348109cde9dae307215fc (127.5.83.67:45985)
I20260923 20:20:09.499555 22623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:55977
I20260923 20:20:09.507387 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:20:09.524608 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.512191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.5.83.126:36425)
0923 20:20:09.512417 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:09.512427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:09.512888 (+ 461us) server_negotiation.cc:407] Connection header received
0923 20:20:09.513031 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:09.513035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:09.513100 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:09.513183 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:09.514590 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.515366 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.516665 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.517072 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.517176 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.519664 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:09.519669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:09.519672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:09.519703 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:09.522198 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:09.522822 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:09.522826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:09.522828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:09.522950 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:09.523288 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:09.523291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:09.523293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:09.523599 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:09.523788 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:09.524129 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 20:20:09.524312 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:20:09.528993 22623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:20:09.535925 22623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:20:09.572897 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.548931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:34199 (local address 127.0.0.1:54604)
0923 20:20:09.561513 (+ 12582us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:09.561583 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:09.561862 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:09.563641 (+ 1779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:09.563653 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:09.563694 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:09.564177 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:09.564194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.566806 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.566814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:09.569965 (+ 3151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:09.569978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.571847 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.571854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:09.572055 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.572115 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 20:20:09.572292 (+ 177us) client_negotiation.cc:240] Negotiation successful
0923 20:20:09.572499 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12240,"thread_start_us":13700,"threads_started":1}
I20260923 20:20:09.572897 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.549217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.5.83.65:34199)
0923 20:20:09.562654 (+ 13437us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:09.562664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:09.562694 (+ 30us) server_negotiation.cc:407] Connection header received
0923 20:20:09.562806 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:09.562814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:09.563093 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:09.563279 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:09.564450 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.566601 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.570163 (+ 3562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.571629 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.571803 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.572046 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:09.572326 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 20:20:09.572497 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13225,"thread_start_us":13024,"threads_started":1}
I20260923 20:20:09.574162 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.549194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:45985 (local address 127.0.0.1:42036)
0923 20:20:09.562864 (+ 13670us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:09.562879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:09.563047 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:09.564346 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:09.564354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:09.564389 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:09.564918 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:09.564936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.567723 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.567732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:09.571142 (+ 3410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:09.571155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.573317 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.573324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:09.573510 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.573555 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 20:20:09.573637 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 20:20:09.573792 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13556,"thread_start_us":11819,"threads_started":1}
I20260923 20:20:09.574370 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.549562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.5.83.67:45985)
0923 20:20:09.560549 (+ 10987us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:09.560564 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:09.562998 (+ 2434us) server_negotiation.cc:407] Connection header received
0923 20:20:09.563302 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:09.563311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:09.563574 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:09.563785 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:09.565185 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.567437 (+ 2252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.571364 (+ 3927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.573116 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.573254 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.573487 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:09.573825 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 20:20:09.574007 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10708,"thread_start_us":11004,"threads_started":1}
I20260923 20:20:09.577719 23015 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2f591d9555467ba26528f53e60dfca (DEFAULT_TABLE table=test-table [id=e5c11c734f5b4379a2b31d16413ad19b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:09.577772 22747 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2f591d9555467ba26528f53e60dfca (DEFAULT_TABLE table=test-table [id=e5c11c734f5b4379a2b31d16413ad19b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:09.579277 22747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2f591d9555467ba26528f53e60dfca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:09.579295 23015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2f591d9555467ba26528f53e60dfca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:09.580278 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.563256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:38449 (local address 127.0.0.1:44304)
0923 20:20:09.569850 (+ 6594us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:09.569869 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:09.569984 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:09.571253 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:09.571259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:09.571293 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:09.571692 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:09.571703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.575145 (+ 3442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.575151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:09.577451 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:09.577468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.579384 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.579392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:09.579583 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.579625 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 20:20:09.579733 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:20:09.579894 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6466,"thread_start_us":6781,"threads_started":1}
I20260923 20:20:09.580302 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.563337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.5.83.66:38449)
0923 20:20:09.570658 (+ 7321us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:09.570667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:09.570695 (+ 28us) server_negotiation.cc:407] Connection header received
0923 20:20:09.570800 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:09.570808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:09.571037 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:09.571245 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:09.571931 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.574977 (+ 3046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.577646 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.579149 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.579297 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.579489 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:09.579799 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 20:20:09.579966 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7112,"spinlock_wait_cycles":23936,"thread_start_us":7082,"threads_started":1}
I20260923 20:20:09.584263 22881 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2f591d9555467ba26528f53e60dfca (DEFAULT_TABLE table=test-table [id=e5c11c734f5b4379a2b31d16413ad19b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:09.585729 22881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2f591d9555467ba26528f53e60dfca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:09.592932 23099 tablet_bootstrap.cc:492] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10: Bootstrap starting.
I20260923 20:20:09.597415 23100 tablet_bootstrap.cc:492] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc: Bootstrap starting.
I20260923 20:20:09.597500 23099 tablet_bootstrap.cc:654] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:09.599051 23099 log.cc:824] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:09.601657 23102 tablet_bootstrap.cc:492] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c: Bootstrap starting.
I20260923 20:20:09.601788 23100 tablet_bootstrap.cc:654] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:09.602674 23099 tablet_bootstrap.cc:492] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10: No bootstrap required, opened a new log
I20260923 20:20:09.602983 23099 ts_tablet_manager.cc:1397] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260923 20:20:09.603190 23100 log.cc:824] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:09.604736 23102 tablet_bootstrap.cc:654] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:09.605861 23102 log.cc:824] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:09.606528 23100 tablet_bootstrap.cc:492] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc: No bootstrap required, opened a new log
I20260923 20:20:09.606886 23100 ts_tablet_manager.cc:1397] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260923 20:20:09.608402 23102 tablet_bootstrap.cc:492] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c: No bootstrap required, opened a new log
I20260923 20:20:09.608717 23102 ts_tablet_manager.cc:1397] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260923 20:20:09.615653 23102 raft_consensus.cc:348] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34199 } } peers { permanent_uuid: "46849de13f0348109cde9dae307215fc" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45985 } } peers { permanent_uuid: "2ef52514966c4a2595746055ea76512c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38449 } }
I20260923 20:20:09.615720 23099 raft_consensus.cc:348] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34199 } } peers { permanent_uuid: "46849de13f0348109cde9dae307215fc" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45985 } } peers { permanent_uuid: "2ef52514966c4a2595746055ea76512c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38449 } }
I20260923 20:20:09.616238 23099 raft_consensus.cc:374] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:09.616297 23102 raft_consensus.cc:374] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:09.616518 23102 raft_consensus.cc:729] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef52514966c4a2595746055ea76512c, State: Initialized, Role: FOLLOWER
I20260923 20:20:09.616416 23099 raft_consensus.cc:729] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0196003bce48bf8d03a7ef5b0dfb10, State: Initialized, Role: FOLLOWER
I20260923 20:20:09.617223 23102 consensus_queue.cc:260] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34199 } } peers { permanent_uuid: "46849de13f0348109cde9dae307215fc" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45985 } } peers { permanent_uuid: "2ef52514966c4a2595746055ea76512c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38449 } }
I20260923 20:20:09.617759 23099 consensus_queue.cc:260] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34199 } } peers { permanent_uuid: "46849de13f0348109cde9dae307215fc" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45985 } } peers { permanent_uuid: "2ef52514966c4a2595746055ea76512c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38449 } }
I20260923 20:20:09.618464 22946 heartbeater.cc:503] Master 127.5.83.126:36425 was elected leader, sending a full tablet report...
I20260923 20:20:09.618757 23100 raft_consensus.cc:348] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34199 } } peers { permanent_uuid: "46849de13f0348109cde9dae307215fc" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45985 } } peers { permanent_uuid: "2ef52514966c4a2595746055ea76512c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38449 } }
I20260923 20:20:09.619210 23100 raft_consensus.cc:374] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:09.619313 23100 raft_consensus.cc:729] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46849de13f0348109cde9dae307215fc, State: Initialized, Role: FOLLOWER
I20260923 20:20:09.620033 23099 ts_tablet_manager.cc:1428] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20260923 20:20:09.619915 23100 consensus_queue.cc:260] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34199 } } peers { permanent_uuid: "46849de13f0348109cde9dae307215fc" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45985 } } peers { permanent_uuid: "2ef52514966c4a2595746055ea76512c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38449 } }
I20260923 20:20:09.621063 23102 ts_tablet_manager.cc:1428] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c: Time spent starting tablet: real 0.012s user 0.009s sys 0.001s
I20260923 20:20:09.622170 23080 heartbeater.cc:503] Master 127.5.83.126:36425 was elected leader, sending a full tablet report...
I20260923 20:20:09.622866 23100 ts_tablet_manager.cc:1428] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260923 20:20:09.643581 23105 raft_consensus.cc:482] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:20:09.643899 23105 raft_consensus.cc:504] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34199 } } peers { permanent_uuid: "46849de13f0348109cde9dae307215fc" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45985 } } peers { permanent_uuid: "2ef52514966c4a2595746055ea76512c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38449 } }
I20260923 20:20:09.645702 23105 leader_election.cc:288] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae0196003bce48bf8d03a7ef5b0dfb10 (127.5.83.65:34199), 46849de13f0348109cde9dae307215fc (127.5.83.67:45985)
I20260923 20:20:09.652815 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.646258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:39601 (local address 127.5.83.65:34199)
0923 20:20:09.646623 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:09.646634 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:09.647380 (+ 746us) server_negotiation.cc:407] Connection header received
0923 20:20:09.647585 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:09.647591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:09.647665 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:09.647750 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:09.648290 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.649191 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.651022 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.651796 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.651942 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.652067 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:09.652402 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 20:20:09.652558 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:20:09.652997 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.646037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:34199 (local address 127.5.83.66:39601)
0923 20:20:09.647247 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:09.647270 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:09.647433 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:09.647774 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:09.647778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:09.647800 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:09.648148 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:09.648161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.649316 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.649324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:09.650857 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:09.650868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.652020 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.652025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:09.652180 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.652252 (+ 72us) client_negotiation.cc:769] Sending connection context
0923 20:20:09.652368 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 20:20:09.652509 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":83,"thread_start_us":1080,"threads_started":1}
I20260923 20:20:09.653065 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.646182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:45985 (local address 127.5.83.66:37905)
0923 20:20:09.647458 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:09.647476 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:09.647632 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:09.648068 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:09.648072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:09.648104 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:09.648447 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:09.648455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.649648 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.649654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:09.651037 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:09.651049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.652187 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.652193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:09.652399 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.652422 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:20:09.652529 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 20:20:09.652665 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"mutex_wait_us":2,"thread_start_us":647,"threads_started":1}
I20260923 20:20:09.653148 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.646370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:37905 (local address 127.5.83.67:45985)
0923 20:20:09.646620 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:09.646629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:09.647598 (+ 969us) server_negotiation.cc:407] Connection header received
0923 20:20:09.647823 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:09.647827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:09.647942 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:09.648048 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:09.648662 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.649509 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.651190 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.651973 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.652124 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.652233 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:09.652600 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 20:20:09.652722 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:20:09.654104 22767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2f591d9555467ba26528f53e60dfca" candidate_uuid: "2ef52514966c4a2595746055ea76512c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10" is_pre_election: true
I20260923 20:20:09.654127 23035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2f591d9555467ba26528f53e60dfca" candidate_uuid: "2ef52514966c4a2595746055ea76512c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46849de13f0348109cde9dae307215fc" is_pre_election: true
I20260923 20:20:09.654637 22767 raft_consensus.cc:2454] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef52514966c4a2595746055ea76512c in term 0.
I20260923 20:20:09.654636 23035 raft_consensus.cc:2454] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef52514966c4a2595746055ea76512c in term 0.
I20260923 20:20:09.655350 22835 leader_election.cc:302] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef52514966c4a2595746055ea76512c, 46849de13f0348109cde9dae307215fc; no voters:
I20260923 20:20:09.655800 23105 raft_consensus.cc:2781] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:20:09.655964 23105 raft_consensus.cc:482] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:20:09.656088 23105 raft_consensus.cc:3037] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:09.658409 23105 raft_consensus.cc:504] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34199 } } peers { permanent_uuid: "46849de13f0348109cde9dae307215fc" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45985 } } peers { permanent_uuid: "2ef52514966c4a2595746055ea76512c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38449 } }
I20260923 20:20:09.659142 23105 leader_election.cc:288] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [CANDIDATE]: Term 1 election: Requested vote from peers ae0196003bce48bf8d03a7ef5b0dfb10 (127.5.83.65:34199), 46849de13f0348109cde9dae307215fc (127.5.83.67:45985)
I20260923 20:20:09.659588 22767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2f591d9555467ba26528f53e60dfca" candidate_uuid: "2ef52514966c4a2595746055ea76512c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10"
I20260923 20:20:09.659710 23035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f2f591d9555467ba26528f53e60dfca" candidate_uuid: "2ef52514966c4a2595746055ea76512c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46849de13f0348109cde9dae307215fc"
I20260923 20:20:09.659822 22767 raft_consensus.cc:3037] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:09.659919 23035 raft_consensus.cc:3037] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:09.663244 23035 raft_consensus.cc:2454] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef52514966c4a2595746055ea76512c in term 1.
I20260923 20:20:09.663252 22767 raft_consensus.cc:2454] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef52514966c4a2595746055ea76512c in term 1.
I20260923 20:20:09.663715 22835 leader_election.cc:302] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef52514966c4a2595746055ea76512c, 46849de13f0348109cde9dae307215fc; no voters:
I20260923 20:20:09.664062 23105 raft_consensus.cc:2781] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:09.664520 23105 raft_consensus.cc:686] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 1 LEADER]: Becoming Leader. State: Replica: 2ef52514966c4a2595746055ea76512c, State: Running, Role: LEADER
I20260923 20:20:09.665091 23105 consensus_queue.cc:237] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34199 } } peers { permanent_uuid: "46849de13f0348109cde9dae307215fc" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45985 } } peers { permanent_uuid: "2ef52514966c4a2595746055ea76512c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38449 } }
I20260923 20:20:09.670392 22621 catalog_manager.cc:5495] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ef52514966c4a2595746055ea76512c (127.5.83.66). New cstate: current_term: 1 leader_uuid: "2ef52514966c4a2595746055ea76512c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46849de13f0348109cde9dae307215fc" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef52514966c4a2595746055ea76512c" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 38449 } health_report { overall_health: HEALTHY } } }
W20260923 20:20:09.674461 22813 tablet.cc:2369] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:20:09.694123 22947 tablet.cc:2369] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:20:09.703177 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.697653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.5.83.66:38449)
0923 20:20:09.697866 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:09.697872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:09.697983 (+ 111us) server_negotiation.cc:407] Connection header received
0923 20:20:09.698264 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:09.698269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:09.698343 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:09.698501 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:20:09.698884 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.699617 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.701060 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.701480 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.701614 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.702092 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:20:09.702591 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:20:09.702761 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:09.702825 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 20:20:09.702904 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:20:09.711118 23035 raft_consensus.cc:1261] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc [term 1 FOLLOWER]: Refusing update from remote peer 2ef52514966c4a2595746055ea76512c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:20:09.711117 22767 raft_consensus.cc:1261] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10 [term 1 FOLLOWER]: Refusing update from remote peer 2ef52514966c4a2595746055ea76512c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:20:09.712100 23105 consensus_queue.cc:1035] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae0196003bce48bf8d03a7ef5b0dfb10" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 34199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:20:09.712384 23110 consensus_queue.cc:1035] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [LEADER]: Connected to new peer: Peer: permanent_uuid: "46849de13f0348109cde9dae307215fc" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:20:09.739571 22621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:20:09.739885 22621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:20:09.743378 22621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 780f79d1de9c4a87aa139bfdb750d738: Sending DeleteTablet for 3 replicas of tablet 5f2f591d9555467ba26528f53e60dfca
I20260923 20:20:09.744642 23015 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2f591d9555467ba26528f53e60dfca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:09 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:42036
I20260923 20:20:09.744684 22881 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2f591d9555467ba26528f53e60dfca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:09 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:44304
I20260923 20:20:09.745750 22747 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2f591d9555467ba26528f53e60dfca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:09 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:54604
I20260923 20:20:09.747511 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 22684
I20260923 20:20:09.775156 23125 tablet_replica.cc:330] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10: stopping tablet replica
I20260923 20:20:09.777170 23125 raft_consensus.cc:2229] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:20:09.779640 23125 raft_consensus.cc:2258] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:09.785212 23124 tablet_replica.cc:330] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc: stopping tablet replica
I20260923 20:20:09.785625 23124 raft_consensus.cc:2229] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:20:09.786163 23124 raft_consensus.cc:2258] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:09.787748 23125 ts_tablet_manager.cc:1905] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:09.789229 23123 tablet_replica.cc:330] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c: stopping tablet replica
I20260923 20:20:09.789742 23123 raft_consensus.cc:2229] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:20:09.790889 23124 ts_tablet_manager.cc:1905] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:09.790920 23123 raft_consensus.cc:2258] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:09.792685 23123 ts_tablet_manager.cc:1905] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 20:20:09 dist-test-slave-vl24 krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:20:09.798100 23124 ts_tablet_manager.cc:1918] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:09.798358 23124 log.cc:1196] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/5f2f591d9555467ba26528f53e60dfca
I20260923 20:20:09.798966 23124 ts_tablet_manager.cc:1939] T 5f2f591d9555467ba26528f53e60dfca P 46849de13f0348109cde9dae307215fc: Deleting consensus metadata
I20260923 20:20:09.799991 23123 ts_tablet_manager.cc:1918] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:09.800108 23125 ts_tablet_manager.cc:1918] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:09.800226 23123 log.cc:1196] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/5f2f591d9555467ba26528f53e60dfca
I20260923 20:20:09.801074 22608 catalog_manager.cc:4841] TS 46849de13f0348109cde9dae307215fc (127.5.83.67:45985): tablet 5f2f591d9555467ba26528f53e60dfca (table test-table [id=e5c11c734f5b4379a2b31d16413ad19b]) successfully deleted
I20260923 20:20:09.801512 23123 ts_tablet_manager.cc:1939] T 5f2f591d9555467ba26528f53e60dfca P 2ef52514966c4a2595746055ea76512c: Deleting consensus metadata
I20260923 20:20:09.803231 22609 catalog_manager.cc:4841] TS 2ef52514966c4a2595746055ea76512c (127.5.83.66:38449): tablet 5f2f591d9555467ba26528f53e60dfca (table test-table [id=e5c11c734f5b4379a2b31d16413ad19b]) successfully deleted
I20260923 20:20:09.803799 23125 log.cc:1196] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/5f2f591d9555467ba26528f53e60dfca
I20260923 20:20:09.804458 23125 ts_tablet_manager.cc:1939] T 5f2f591d9555467ba26528f53e60dfca P ae0196003bce48bf8d03a7ef5b0dfb10: Deleting consensus metadata
I20260923 20:20:09.806289 22609 catalog_manager.cc:4841] TS ae0196003bce48bf8d03a7ef5b0dfb10 (127.5.83.65:34199): tablet 5f2f591d9555467ba26528f53e60dfca (table test-table [id=e5c11c734f5b4379a2b31d16413ad19b]) successfully deleted
I20260923 20:20:09.809360 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.749301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.5.83.65:34199)
0923 20:20:09.749532 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:09.749541 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:09.774024 (+ 24483us) server_negotiation.cc:407] Connection header received
0923 20:20:09.775493 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:09.775503 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:09.775623 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:09.775722 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:09.780264 (+ 4542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.781843 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.788702 (+ 6859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.789381 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.789511 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.802226 (+ 12715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:09.802240 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:09.802257 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:09.802301 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:09.806109 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:09.806992 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:09.807003 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:09.807007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:09.807166 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:09.807605 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:09.807611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:09.807614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:09.808119 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:09.808426 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:09.808762 (+ 336us) server_negotiation.cc:299] Negotiation successful
0923 20:20:09.808989 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:20:09.810163 22807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:09.905347 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 22684
I20260923 20:20:09.924575 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 22818
Sep 23 20:20:09 dist-test-slave-vl24 krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:20:09.943463 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:09.925371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.5.83.66:38449)
0923 20:20:09.925566 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:09.925571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:09.925593 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:20:09.925729 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:09.925736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:09.925846 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:09.925962 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:09.927148 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.928433 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.929810 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:09.930455 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:09.930585 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:09.935261 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:09.935274 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:09.935290 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:09.935330 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:09.939204 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:09.940144 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:09.940159 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:09.940163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:09.940346 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:09.940818 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:09.940823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:09.940826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:09.941294 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:09.941641 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:09.942819 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0923 20:20:09.943049 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 20:20:09.944205 22941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:10.061335 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 22818
I20260923 20:20:10.077463 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 22952
Sep 23 20:20:10 dist-test-slave-vl24 krb5kdc[22578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:20:10.092742 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:10.078259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.5.83.67:45985)
0923 20:20:10.078453 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:10.078457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:10.078475 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:20:10.078584 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:10.078588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:10.078657 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:10.078740 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:10.080046 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:10.080843 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:10.082137 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:10.082550 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:10.082659 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:10.087210 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:10.087228 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:10.087244 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:10.087281 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:10.090054 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:10.090702 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:10.090710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:10.090712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:10.090843 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:10.091220 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:10.091225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:10.091227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:10.091644 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:10.091876 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:10.092198 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 20:20:10.092385 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 20:20:10.093428 23075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:10.183578 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 22952
I20260923 20:20:10.200448 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 22591
I20260923 20:20:10.201550 22653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:10.314319 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 22591
2026-09-23T20:20:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4364 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T20:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:10Z Disabled control of system clock
I20260923 20:20:10.398273 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:41475
--webserver_interface=127.5.83.126
--webserver_port=0
--builtin_ntp_servers=127.5.83.84:37901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:41475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:20:10.737419 23134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:10.737721 23134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:10.737807 23134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:10.745188 23134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:10.745304 23134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:10.745343 23134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:10.745401 23134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:10.745447 23134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:10.754745 23134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:37901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:41475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:10.756013 23134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:10.757786 23134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:10.769681 23139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:10.769618 23140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:10.769659 23142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:10.770483 23134 server_base.cc:1034] running on GCE node
I20260923 20:20:10.771221 23134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:10.772634 23134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:10.773867 23134 hybrid_clock.cc:648] HybridClock initialized: now 1790194810773821 us; error 55 us; skew 500 ppm
I20260923 20:20:10.774223 23134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:20:10.776615 23134 webserver.cc:460] Webserver started at http://127.5.83.126:36603/ using document root <none> and password file <none>
I20260923 20:20:10.777294 23134 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:10.777428 23134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:10.777748 23134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:10.780718 23134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "5993c85910ee42e7b08fc3e42c943479"
format_stamp: "Formatted at 2026-09-23 20:20:10 on dist-test-slave-vl24"
server_key: "c4e2600c8af78087b53f103fe3f56f7a"
server_key_iv: "e4986a09d6d0e690404a228aeed41c30"
server_key_version: "encryptionkey@0"
I20260923 20:20:10.781601 23134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "5993c85910ee42e7b08fc3e42c943479"
format_stamp: "Formatted at 2026-09-23 20:20:10 on dist-test-slave-vl24"
server_key: "c4e2600c8af78087b53f103fe3f56f7a"
server_key_iv: "e4986a09d6d0e690404a228aeed41c30"
server_key_version: "encryptionkey@0"
I20260923 20:20:10.786700 23134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260923 20:20:10.790086 23148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:10.791385 23134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:20:10.791571 23134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "5993c85910ee42e7b08fc3e42c943479"
format_stamp: "Formatted at 2026-09-23 20:20:10 on dist-test-slave-vl24"
server_key: "c4e2600c8af78087b53f103fe3f56f7a"
server_key_iv: "e4986a09d6d0e690404a228aeed41c30"
server_key_version: "encryptionkey@0"
I20260923 20:20:10.791786 23134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:10.809623 23134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:10.810549 23134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:10.810870 23134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:10.829654 23134 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:41475
I20260923 20:20:10.829698 23199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:41475 every 8 connection(s)
I20260923 20:20:10.831043 23134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:10.834213 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 23134
I20260923 20:20:10.834388 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:10.834827 5453 external_mini_cluster.cc:1428] Setting key eec84a26a0ddaaad9f153a15c9df4550
I20260923 20:20:10.834969 23200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:10.847601 23200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479: Bootstrap starting.
I20260923 20:20:10.848464 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:10.836795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.5.83.126:41475)
0923 20:20:10.837766 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:10.837777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:10.837820 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:20:10.839712 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:10.839734 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:10.840204 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:10.840652 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:10.841237 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:10.842534 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:10.843930 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:10.844692 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:10.844909 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:10.846134 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:10.846142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:10.846157 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:10.846716 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:10.846798 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:10.846818 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:10.846939 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:10.847170 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 20:20:10.847477 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":343,"threads_started":1}
I20260923 20:20:10.850904 23200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:10.852070 23200 log.cc:824] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:10.855140 23200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479: No bootstrap required, opened a new log
I20260923 20:20:10.863006 23200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5993c85910ee42e7b08fc3e42c943479" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41475 } }
I20260923 20:20:10.863461 23200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:10.863579 23200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5993c85910ee42e7b08fc3e42c943479, State: Initialized, Role: FOLLOWER
I20260923 20:20:10.864162 23200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5993c85910ee42e7b08fc3e42c943479" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41475 } }
I20260923 20:20:10.864419 23200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:10.864583 23200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:10.864779 23200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:10.867089 23200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5993c85910ee42e7b08fc3e42c943479" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41475 } }
I20260923 20:20:10.867714 23200 leader_election.cc:302] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5993c85910ee42e7b08fc3e42c943479; no voters:
I20260923 20:20:10.868466 23200 leader_election.cc:288] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:10.868618 23205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:10.869375 23205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [term 1 LEADER]: Becoming Leader. State: Replica: 5993c85910ee42e7b08fc3e42c943479, State: Running, Role: LEADER
I20260923 20:20:10.869935 23205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5993c85910ee42e7b08fc3e42c943479" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41475 } }
I20260923 20:20:10.870831 23200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:10.872885 23207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5993c85910ee42e7b08fc3e42c943479. Latest consensus state: current_term: 1 leader_uuid: "5993c85910ee42e7b08fc3e42c943479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5993c85910ee42e7b08fc3e42c943479" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41475 } } }
I20260923 20:20:10.873298 23207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:10.878396 23206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5993c85910ee42e7b08fc3e42c943479" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5993c85910ee42e7b08fc3e42c943479" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 41475 } } }
I20260923 20:20:10.878677 23206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:10.879910 23211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:10.885459 23211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:10.895172 23211 catalog_manager.cc:1347] Generated new cluster ID: 9e38e6395702458bb938e3cba45128ba
I20260923 20:20:10.895340 23211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:10.925154 23211 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:10.926503 23211 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:10.934947 23211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479: Generated new TSK 0
I20260923 20:20:10.935811 23211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:10.951207 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41475
--builtin_ntp_servers=127.5.83.84:37901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:20:11.300372 23224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:11.300788 23224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:11.300979 23224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:11.308380 23224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:11.308526 23224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:11.308729 23224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:20:11.318408 23224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:37901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:11.319892 23224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:11.321821 23224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:11.335464 23229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:11.335520 23230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:11.335439 23232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:11.336663 23224 server_base.cc:1034] running on GCE node
I20260923 20:20:11.337311 23224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:11.338343 23224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:11.339604 23224 hybrid_clock.cc:648] HybridClock initialized: now 1790194811339550 us; error 45 us; skew 500 ppm
I20260923 20:20:11.339978 23224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:20:11.342363 23224 webserver.cc:460] Webserver started at http://127.5.83.65:41459/ using document root <none> and password file <none>
I20260923 20:20:11.343075 23224 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:11.343173 23224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:11.343454 23224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:11.346458 23224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "04d9ec181f3c49ca84599c93065f4e67"
format_stamp: "Formatted at 2026-09-23 20:20:11 on dist-test-slave-vl24"
server_key: "963124a0b9416b537ea65bea522d4619"
server_key_iv: "7ecd7ad06463b47822088b1a50a50ef0"
server_key_version: "encryptionkey@0"
I20260923 20:20:11.347216 23224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "04d9ec181f3c49ca84599c93065f4e67"
format_stamp: "Formatted at 2026-09-23 20:20:11 on dist-test-slave-vl24"
server_key: "963124a0b9416b537ea65bea522d4619"
server_key_iv: "7ecd7ad06463b47822088b1a50a50ef0"
server_key_version: "encryptionkey@0"
I20260923 20:20:11.352458 23224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:20:11.355938 23238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:11.357285 23224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:20:11.357416 23224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "04d9ec181f3c49ca84599c93065f4e67"
format_stamp: "Formatted at 2026-09-23 20:20:11 on dist-test-slave-vl24"
server_key: "963124a0b9416b537ea65bea522d4619"
server_key_iv: "7ecd7ad06463b47822088b1a50a50ef0"
server_key_version: "encryptionkey@0"
I20260923 20:20:11.357599 23224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:11.386137 23224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:11.387022 23224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:11.387262 23224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:11.388219 23224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:11.389762 23224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:11.389848 23224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:11.389935 23224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:11.390015 23224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:11.425820 23224 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:46443
I20260923 20:20:11.425864 23350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:46443 every 8 connection(s)
I20260923 20:20:11.427100 23224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:20:11.431547 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 23224
I20260923 20:20:11.431727 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:20:11.432215 5453 external_mini_cluster.cc:1428] Setting key bc1b0e8a936b4179548c71c078076c33
I20260923 20:20:11.435259 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41475
--builtin_ntp_servers=127.5.83.84:37901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:20:11.439731 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:11.429553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:49153 (local address 127.5.83.126:41475)
0923 20:20:11.429986 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:11.429990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:11.431101 (+ 1111us) server_negotiation.cc:407] Connection header received
0923 20:20:11.433185 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:11.433189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:11.433295 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:11.433420 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:11.434568 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:11.435294 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:11.436915 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:11.437372 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:11.437503 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:11.438873 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:11.438878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:11.438881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:11.438900 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:11.438938 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:11.438950 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:11.439048 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:11.439268 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 20:20:11.439366 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":226,"threads_started":1}
I20260923 20:20:11.440797 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:11.429973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41475 (local address 127.5.83.65:49153)
0923 20:20:11.430950 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:11.430986 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:11.432869 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:11.433619 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:11.433627 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:11.433731 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:11.434391 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:11.434408 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:11.435469 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:11.435474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:11.436773 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:11.436784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:11.437603 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:11.437611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:11.437842 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:11.438553 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:20:11.438556 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:11.438598 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:20:11.438673 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:11.439048 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:11.439062 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:11.439072 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:20:11.439264 (+ 192us) client_negotiation.cc:240] Negotiation successful
0923 20:20:11.439651 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":274,"threads_started":1}
I20260923 20:20:11.442348 23351 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41475
I20260923 20:20:11.442654 23351 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:11.443387 23351 heartbeater.cc:511] Master 127.5.83.126:41475 requested a full tablet report, sending...
I20260923 20:20:11.445448 23165 ts_manager.cc:194] Registered new tserver with Master: 04d9ec181f3c49ca84599c93065f4e67 (127.5.83.65:46443)
I20260923 20:20:11.446934 23165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.65:49153
W20260923 20:20:11.766738 23355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:11.767079 23355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:11.767247 23355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:11.774981 23355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:11.775125 23355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:11.775300 23355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:20:11.784972 23355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:37901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:11.786301 23355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:11.788084 23355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:11.801240 23360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:11.801285 23361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:11.801311 23363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:11.802052 23355 server_base.cc:1034] running on GCE node
I20260923 20:20:11.802695 23355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:11.803654 23355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:11.804880 23355 hybrid_clock.cc:648] HybridClock initialized: now 1790194811804835 us; error 59 us; skew 500 ppm
I20260923 20:20:11.805204 23355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:20:11.807588 23355 webserver.cc:460] Webserver started at http://127.5.83.66:46675/ using document root <none> and password file <none>
I20260923 20:20:11.808301 23355 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:11.808441 23355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:11.808827 23355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:11.811779 23355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "d24bab6a180e4d3ea178d76af4c11ec8"
format_stamp: "Formatted at 2026-09-23 20:20:11 on dist-test-slave-vl24"
server_key: "35e26f7ebbcbe44f4273f23786e85f9e"
server_key_iv: "6301ba8bddefb11ddfcae27f969e1fa8"
server_key_version: "encryptionkey@0"
I20260923 20:20:11.812640 23355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "d24bab6a180e4d3ea178d76af4c11ec8"
format_stamp: "Formatted at 2026-09-23 20:20:11 on dist-test-slave-vl24"
server_key: "35e26f7ebbcbe44f4273f23786e85f9e"
server_key_iv: "6301ba8bddefb11ddfcae27f969e1fa8"
server_key_version: "encryptionkey@0"
I20260923 20:20:11.817597 23355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:20:11.821055 23369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:11.822372 23355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:20:11.822551 23355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "d24bab6a180e4d3ea178d76af4c11ec8"
format_stamp: "Formatted at 2026-09-23 20:20:11 on dist-test-slave-vl24"
server_key: "35e26f7ebbcbe44f4273f23786e85f9e"
server_key_iv: "6301ba8bddefb11ddfcae27f969e1fa8"
server_key_version: "encryptionkey@0"
I20260923 20:20:11.822764 23355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:11.863983 23355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:11.864938 23355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:11.865227 23355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:11.866254 23355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:11.867820 23355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:11.867945 23355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:11.868079 23355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:11.868170 23355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:11.901825 23355 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:37929
I20260923 20:20:11.901873 23481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:37929 every 8 connection(s)
I20260923 20:20:11.903158 23355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:20:11.912847 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 23355
I20260923 20:20:11.915475 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:11.905625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:34081 (local address 127.5.83.126:41475)
0923 20:20:11.905804 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:11.905808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:11.907192 (+ 1384us) server_negotiation.cc:407] Connection header received
0923 20:20:11.909014 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:11.909018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:11.909098 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:11.909188 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:11.910219 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:11.910947 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:11.912406 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:11.912901 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:11.913060 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:11.914753 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:11.914757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:11.914761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:11.914779 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:11.914810 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:11.914821 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:11.914891 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:11.915164 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 20:20:11.915243 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:20:11.917228 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:11.906168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41475 (local address 127.5.83.66:34081)
0923 20:20:11.907062 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:11.907098 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:11.908698 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:11.909379 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:11.909388 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:11.909474 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:11.910057 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:11.910073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:11.911070 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:11.911073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:11.912279 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:11.912290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:11.913209 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:11.913217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:11.913559 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:11.914355 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:20:11.914359 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:11.914427 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:20:11.914550 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:11.914922 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:11.914935 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:11.914945 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:20:11.915152 (+ 207us) client_negotiation.cc:240] Negotiation successful
0923 20:20:11.915621 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":293,"threads_started":1}
I20260923 20:20:11.917608 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:20:11.918097 5453 external_mini_cluster.cc:1428] Setting key 1fc8455491e1ce656859d81dacc275b4
I20260923 20:20:11.918776 23482 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41475
I20260923 20:20:11.919148 23482 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:11.920037 23482 heartbeater.cc:511] Master 127.5.83.126:41475 requested a full tablet report, sending...
I20260923 20:20:11.921023 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41475
--builtin_ntp_servers=127.5.83.84:37901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:20:11.921566 23165 ts_manager.cc:194] Registered new tserver with Master: d24bab6a180e4d3ea178d76af4c11ec8 (127.5.83.66:37929)
I20260923 20:20:11.922380 23165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.66:34081
W20260923 20:20:12.257033 23485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:12.257318 23485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:12.257469 23485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:12.264781 23485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:12.264889 23485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:12.265033 23485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:20:12.275085 23485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:37901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:41475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:12.276378 23485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:12.278144 23485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:12.291034 23490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:12.290993 23493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:12.291085 23491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:12.291774 23485 server_base.cc:1034] running on GCE node
I20260923 20:20:12.292371 23485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:12.293358 23485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:12.294562 23485 hybrid_clock.cc:648] HybridClock initialized: now 1790194812294526 us; error 34 us; skew 500 ppm
I20260923 20:20:12.294898 23485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:20:12.297384 23485 webserver.cc:460] Webserver started at http://127.5.83.67:35817/ using document root <none> and password file <none>
I20260923 20:20:12.298085 23485 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:12.298199 23485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:12.298547 23485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:12.301463 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "7d8b704293004b58b71aa472082e5791"
format_stamp: "Formatted at 2026-09-23 20:20:12 on dist-test-slave-vl24"
server_key: "e8d827e884f7f88f71381d636c2e797d"
server_key_iv: "8b5c8611a1881ca8a688deffb362b6fe"
server_key_version: "encryptionkey@0"
I20260923 20:20:12.302255 23485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "7d8b704293004b58b71aa472082e5791"
format_stamp: "Formatted at 2026-09-23 20:20:12 on dist-test-slave-vl24"
server_key: "e8d827e884f7f88f71381d636c2e797d"
server_key_iv: "8b5c8611a1881ca8a688deffb362b6fe"
server_key_version: "encryptionkey@0"
I20260923 20:20:12.307236 23485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:20:12.310649 23499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:12.311944 23485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260923 20:20:12.312130 23485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "7d8b704293004b58b71aa472082e5791"
format_stamp: "Formatted at 2026-09-23 20:20:12 on dist-test-slave-vl24"
server_key: "e8d827e884f7f88f71381d636c2e797d"
server_key_iv: "8b5c8611a1881ca8a688deffb362b6fe"
server_key_version: "encryptionkey@0"
I20260923 20:20:12.312363 23485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:12.334175 23485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:12.335053 23485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:12.335343 23485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:12.336324 23485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:12.338006 23485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:12.338130 23485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:12.338263 23485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:12.338351 23485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:12.372291 23485 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:45495
I20260923 20:20:12.372383 23611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:45495 every 8 connection(s)
I20260923 20:20:12.373687 23485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:20:12.378142 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 23485
I20260923 20:20:12.378343 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:20:12.378940 5453 external_mini_cluster.cc:1428] Setting key c2f20dc2aeddd2a55b12374946045357
I20260923 20:20:12.387012 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.376368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:46695 (local address 127.5.83.126:41475)
0923 20:20:12.376621 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:12.376626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:12.378130 (+ 1504us) server_negotiation.cc:407] Connection header received
0923 20:20:12.380622 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:12.380628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:12.380726 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:12.380860 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:12.382202 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.382942 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.384442 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.384918 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.385015 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.386312 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:12.386317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:12.386320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:12.386338 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:12.386370 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:12.386381 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:12.386452 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:12.386654 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 20:20:12.386735 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:20:12.388025 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.376941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:41475 (local address 127.5.83.67:46695)
0923 20:20:12.377936 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.378001 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:12.380238 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:12.381111 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:12.381120 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:12.381208 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:12.382036 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:12.382062 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.383072 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.383076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:12.384300 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:12.384312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.385136 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.385141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:12.385344 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.386014 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:20:12.386016 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:12.386056 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:20:12.386135 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:12.386473 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:12.386480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:12.386485 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 20:20:12.386623 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 20:20:12.386909 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":285,"threads_started":1}
I20260923 20:20:12.389433 23612 heartbeater.cc:347] Connected to a master server at 127.5.83.126:41475
I20260923 20:20:12.389755 23612 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:12.390499 23612 heartbeater.cc:511] Master 127.5.83.126:41475 requested a full tablet report, sending...
I20260923 20:20:12.391903 23165 ts_manager.cc:194] Registered new tserver with Master: 7d8b704293004b58b71aa472082e5791 (127.5.83.67:45495)
I20260923 20:20:12.392882 23165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.67:46695
I20260923 20:20:12.394403 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:20:12.404403 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.398881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.5.83.126:41475)
0923 20:20:12.399149 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:12.399157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:12.399458 (+ 301us) server_negotiation.cc:407] Connection header received
0923 20:20:12.399594 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:12.399598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:12.399669 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:12.399750 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:12.400202 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.400971 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.402306 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.402685 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.402777 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.403741 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:12.403745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:12.403748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:12.403764 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:12.403795 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:12.403806 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:12.403873 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:12.403989 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 20:20:12.404055 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:20:12.407874 23165 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:20:12.414574 23165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:20:12.449626 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.428972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:37929 (local address 127.0.0.1:41624)
0923 20:20:12.435361 (+ 6389us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.435411 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:12.436840 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:12.437870 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:12.437879 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:12.437911 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:12.438274 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:12.438287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.440295 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.440300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:12.446151 (+ 5851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:12.446165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.448490 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.448497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:12.448784 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.448851 (+ 67us) client_negotiation.cc:769] Sending connection context
0923 20:20:12.448998 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 20:20:12.449213 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6146,"mutex_wait_us":1,"thread_start_us":5226,"threads_started":1}
I20260923 20:20:12.449800 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.429158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.5.83.66:37929)
0923 20:20:12.437164 (+ 8006us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:12.437172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:12.437201 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:20:12.437302 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:12.437310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:12.437635 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:12.437852 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:12.438442 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.440147 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.446367 (+ 6220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.448257 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.448442 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.448805 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:12.449074 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 20:20:12.449317 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7830,"thread_start_us":7748,"threads_started":1}
I20260923 20:20:12.450099 23351 heartbeater.cc:503] Master 127.5.83.126:41475 was elected leader, sending a full tablet report...
I20260923 20:20:12.453857 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.440948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:46443 (local address 127.0.0.1:56638)
0923 20:20:12.441545 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.441563 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:12.441676 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:12.446772 (+ 5096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:12.446780 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:12.446812 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:12.447129 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:12.447137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.448505 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.448509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:12.451178 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:12.451192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.453077 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.453083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:12.453249 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.453289 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:20:12.453390 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 20:20:12.453544 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":546,"threads_started":1}
I20260923 20:20:12.454072 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.435805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.5.83.65:46443)
0923 20:20:12.446111 (+ 10306us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:12.446118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:12.446145 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:20:12.446227 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:12.446233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:12.446546 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:12.446764 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:12.447292 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.448320 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.451399 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.452878 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.453047 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.453309 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:12.453428 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 20:20:12.453585 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10104,"thread_start_us":9966,"threads_started":1}
I20260923 20:20:12.454002 23417 tablet_service.cc:1467] Processing CreateTablet for tablet a74803512a3e4c7a80a219982143fbd7 (DEFAULT_TABLE table=test-table [id=5fe53a5096d1493a865af81332e535b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:12.455528 23417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74803512a3e4c7a80a219982143fbd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:12.460179 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.427856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:45495 (local address 127.0.0.1:34162)
0923 20:20:12.438145 (+ 10289us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.438174 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:12.438288 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:12.451538 (+ 13250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:12.451545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:12.451578 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:12.451926 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:12.451941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.454099 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.454107 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:12.456020 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:12.456035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.459515 (+ 3480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.459520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:12.459710 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.459746 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 20:20:12.459812 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 20:20:12.459957 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6507,"thread_start_us":7136,"threads_started":1}
I20260923 20:20:12.460170 23286 tablet_service.cc:1467] Processing CreateTablet for tablet a74803512a3e4c7a80a219982143fbd7 (DEFAULT_TABLE table=test-table [id=5fe53a5096d1493a865af81332e535b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:12.461364 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.428185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.5.83.67:45495)
0923 20:20:12.450736 (+ 22551us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:12.450745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:12.450779 (+ 34us) server_negotiation.cc:407] Connection header received
0923 20:20:12.450874 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:12.450882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:12.451132 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:12.452156 (+ 1024us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:12.452415 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.453911 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.456251 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.459288 (+ 3037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.460597 (+ 1309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.460786 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:12.460875 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 20:20:12.461067 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22363,"thread_start_us":23087,"threads_started":1}
I20260923 20:20:12.461695 23286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74803512a3e4c7a80a219982143fbd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:12.464311 23547 tablet_service.cc:1467] Processing CreateTablet for tablet a74803512a3e4c7a80a219982143fbd7 (DEFAULT_TABLE table=test-table [id=5fe53a5096d1493a865af81332e535b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:12.465492 23547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74803512a3e4c7a80a219982143fbd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:12.470248 23630 tablet_bootstrap.cc:492] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8: Bootstrap starting.
I20260923 20:20:12.474071 23630 tablet_bootstrap.cc:654] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:12.475207 23631 tablet_bootstrap.cc:492] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67: Bootstrap starting.
I20260923 20:20:12.475281 23630 log.cc:824] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:12.478614 23631 tablet_bootstrap.cc:654] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:12.478837 23630 tablet_bootstrap.cc:492] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8: No bootstrap required, opened a new log
I20260923 20:20:12.479221 23630 ts_tablet_manager.cc:1397] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260923 20:20:12.480073 23633 tablet_bootstrap.cc:492] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791: Bootstrap starting.
I20260923 20:20:12.480087 23631 log.cc:824] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:12.484717 23633 tablet_bootstrap.cc:654] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:12.486079 23633 log.cc:824] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:12.487259 23631 tablet_bootstrap.cc:492] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67: No bootstrap required, opened a new log
I20260923 20:20:12.487589 23631 ts_tablet_manager.cc:1397] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260923 20:20:12.487757 23630 raft_consensus.cc:348] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37929 } } peers { permanent_uuid: "7d8b704293004b58b71aa472082e5791" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45495 } } peers { permanent_uuid: "04d9ec181f3c49ca84599c93065f4e67" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 46443 } }
I20260923 20:20:12.488225 23630 raft_consensus.cc:374] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:12.488330 23630 raft_consensus.cc:729] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24bab6a180e4d3ea178d76af4c11ec8, State: Initialized, Role: FOLLOWER
I20260923 20:20:12.488848 23633 tablet_bootstrap.cc:492] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791: No bootstrap required, opened a new log
I20260923 20:20:12.489106 23633 ts_tablet_manager.cc:1397] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260923 20:20:12.488977 23630 consensus_queue.cc:260] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37929 } } peers { permanent_uuid: "7d8b704293004b58b71aa472082e5791" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45495 } } peers { permanent_uuid: "04d9ec181f3c49ca84599c93065f4e67" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 46443 } }
I20260923 20:20:12.490140 23482 heartbeater.cc:503] Master 127.5.83.126:41475 was elected leader, sending a full tablet report...
I20260923 20:20:12.490720 23630 ts_tablet_manager.cc:1428] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260923 20:20:12.496683 23631 raft_consensus.cc:348] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37929 } } peers { permanent_uuid: "7d8b704293004b58b71aa472082e5791" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45495 } } peers { permanent_uuid: "04d9ec181f3c49ca84599c93065f4e67" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 46443 } }
I20260923 20:20:12.497205 23631 raft_consensus.cc:374] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:12.497310 23631 raft_consensus.cc:729] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d9ec181f3c49ca84599c93065f4e67, State: Initialized, Role: FOLLOWER
I20260923 20:20:12.497942 23631 consensus_queue.cc:260] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37929 } } peers { permanent_uuid: "7d8b704293004b58b71aa472082e5791" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45495 } } peers { permanent_uuid: "04d9ec181f3c49ca84599c93065f4e67" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 46443 } }
I20260923 20:20:12.499042 23633 raft_consensus.cc:348] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37929 } } peers { permanent_uuid: "7d8b704293004b58b71aa472082e5791" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45495 } } peers { permanent_uuid: "04d9ec181f3c49ca84599c93065f4e67" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 46443 } }
I20260923 20:20:12.499406 23633 raft_consensus.cc:374] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:12.499531 23633 raft_consensus.cc:729] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8b704293004b58b71aa472082e5791, State: Initialized, Role: FOLLOWER
I20260923 20:20:12.499658 23631 ts_tablet_manager.cc:1428] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260923 20:20:12.500144 23633 consensus_queue.cc:260] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37929 } } peers { permanent_uuid: "7d8b704293004b58b71aa472082e5791" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45495 } } peers { permanent_uuid: "04d9ec181f3c49ca84599c93065f4e67" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 46443 } }
I20260923 20:20:12.501271 23612 heartbeater.cc:503] Master 127.5.83.126:41475 was elected leader, sending a full tablet report...
I20260923 20:20:12.501838 23633 ts_tablet_manager.cc:1428] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260923 20:20:12.517705 23636 raft_consensus.cc:482] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:20:12.518015 23636 raft_consensus.cc:504] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37929 } } peers { permanent_uuid: "7d8b704293004b58b71aa472082e5791" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45495 } } peers { permanent_uuid: "04d9ec181f3c49ca84599c93065f4e67" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 46443 } }
I20260923 20:20:12.519573 23636 leader_election.cc:288] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d8b704293004b58b71aa472082e5791 (127.5.83.67:45495), 04d9ec181f3c49ca84599c93065f4e67 (127.5.83.65:46443)
I20260923 20:20:12.526908 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.520935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:59581 (local address 127.5.83.65:46443)
0923 20:20:12.521121 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:12.521129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:12.521527 (+ 398us) server_negotiation.cc:407] Connection header received
0923 20:20:12.521640 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:12.521647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:12.521734 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:12.521828 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:12.522484 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.523249 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.525110 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.525865 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.525962 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.526036 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:12.526516 (+ 480us) server_negotiation.cc:299] Negotiation successful
0923 20:20:12.526655 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:20:12.526983 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.520008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:58713 (local address 127.5.83.67:45495)
0923 20:20:12.520186 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:12.520190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:12.520777 (+ 587us) server_negotiation.cc:407] Connection header received
0923 20:20:12.520995 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:12.521004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:12.521083 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:12.521184 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:12.521889 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.522959 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.525056 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.526077 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.526205 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.526302 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:12.526648 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 20:20:12.526776 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:20:12.527004 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.519931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:45495 (local address 127.5.83.66:58713)
0923 20:20:12.520671 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.520695 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:12.520834 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:12.521219 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:12.521224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:12.521254 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:12.521678 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:12.521692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.523091 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.523097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:12.524861 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:12.524873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.526253 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.526257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:12.526456 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.526493 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 20:20:12.526581 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 20:20:12.526697 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":76,"thread_start_us":385,"threads_started":1}
I20260923 20:20:12.527381 23640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.520779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:46443 (local address 127.5.83.66:59581)
0923 20:20:12.521411 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.521432 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:12.521536 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:12.521888 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:12.521894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:12.521922 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:12.522316 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:12.522328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.523397 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.523401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:12.524892 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:12.524907 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.526076 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.526083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:12.526277 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.526369 (+ 92us) client_negotiation.cc:769] Sending connection context
0923 20:20:12.526443 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 20:20:12.526542 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":5,"thread_start_us":341,"threads_started":1}
I20260923 20:20:12.528090 23567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74803512a3e4c7a80a219982143fbd7" candidate_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d8b704293004b58b71aa472082e5791" is_pre_election: true
I20260923 20:20:12.528445 23306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74803512a3e4c7a80a219982143fbd7" candidate_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d9ec181f3c49ca84599c93065f4e67" is_pre_election: true
I20260923 20:20:12.528772 23567 raft_consensus.cc:2454] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d24bab6a180e4d3ea178d76af4c11ec8 in term 0.
I20260923 20:20:12.529024 23306 raft_consensus.cc:2454] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d24bab6a180e4d3ea178d76af4c11ec8 in term 0.
I20260923 20:20:12.529580 23372 leader_election.cc:302] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d8b704293004b58b71aa472082e5791, d24bab6a180e4d3ea178d76af4c11ec8; no voters:
I20260923 20:20:12.530151 23636 raft_consensus.cc:2781] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:20:12.530313 23636 raft_consensus.cc:482] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:20:12.530443 23636 raft_consensus.cc:3037] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:12.532975 23636 raft_consensus.cc:504] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37929 } } peers { permanent_uuid: "7d8b704293004b58b71aa472082e5791" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45495 } } peers { permanent_uuid: "04d9ec181f3c49ca84599c93065f4e67" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 46443 } }
I20260923 20:20:12.533588 23636 leader_election.cc:288] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [CANDIDATE]: Term 1 election: Requested vote from peers 7d8b704293004b58b71aa472082e5791 (127.5.83.67:45495), 04d9ec181f3c49ca84599c93065f4e67 (127.5.83.65:46443)
I20260923 20:20:12.534060 23567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74803512a3e4c7a80a219982143fbd7" candidate_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d8b704293004b58b71aa472082e5791"
I20260923 20:20:12.534157 23306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74803512a3e4c7a80a219982143fbd7" candidate_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04d9ec181f3c49ca84599c93065f4e67"
I20260923 20:20:12.534282 23567 raft_consensus.cc:3037] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:12.534390 23306 raft_consensus.cc:3037] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:12.536707 23306 raft_consensus.cc:2454] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d24bab6a180e4d3ea178d76af4c11ec8 in term 1.
I20260923 20:20:12.536819 23567 raft_consensus.cc:2454] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d24bab6a180e4d3ea178d76af4c11ec8 in term 1.
I20260923 20:20:12.537104 23372 leader_election.cc:302] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04d9ec181f3c49ca84599c93065f4e67, d24bab6a180e4d3ea178d76af4c11ec8; no voters:
I20260923 20:20:12.537441 23636 raft_consensus.cc:2781] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:12.537882 23636 raft_consensus.cc:686] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 1 LEADER]: Becoming Leader. State: Replica: d24bab6a180e4d3ea178d76af4c11ec8, State: Running, Role: LEADER
I20260923 20:20:12.538367 23636 consensus_queue.cc:237] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37929 } } peers { permanent_uuid: "7d8b704293004b58b71aa472082e5791" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45495 } } peers { permanent_uuid: "04d9ec181f3c49ca84599c93065f4e67" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 46443 } }
I20260923 20:20:12.543118 23165 catalog_manager.cc:5495] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d24bab6a180e4d3ea178d76af4c11ec8 (127.5.83.66). New cstate: current_term: 1 leader_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24bab6a180e4d3ea178d76af4c11ec8" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d8b704293004b58b71aa472082e5791" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04d9ec181f3c49ca84599c93065f4e67" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 46443 } health_report { overall_health: UNKNOWN } } }
I20260923 20:20:12.563228 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.558183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.5.83.66:37929)
0923 20:20:12.558383 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:12.558387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:12.558404 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:20:12.558576 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:12.558580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:12.558672 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:12.558791 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:20:12.559095 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.559772 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.561209 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.561624 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.561777 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.562181 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:20:12.562669 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:20:12.562778 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:12.562858 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 20:20:12.562919 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:20:12.571126 23306 raft_consensus.cc:1261] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67 [term 1 FOLLOWER]: Refusing update from remote peer d24bab6a180e4d3ea178d76af4c11ec8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:20:12.571174 23567 raft_consensus.cc:1261] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791 [term 1 FOLLOWER]: Refusing update from remote peer d24bab6a180e4d3ea178d76af4c11ec8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:20:12.572088 23641 consensus_queue.cc:1035] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04d9ec181f3c49ca84599c93065f4e67" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 46443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:20:12.572499 23636 consensus_queue.cc:1035] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d8b704293004b58b71aa472082e5791" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:20:12.584280 23644 mvcc.cc:205] Tried to move back new op lower bound from 7332637952281886720 to 7332637952161079296. Current Snapshot: MvccSnapshot[applied={T|T < 7332637952281886720}]
I20260923 20:20:12.598289 23164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:20:12.598553 23164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:20:12.602118 23164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5993c85910ee42e7b08fc3e42c943479: Sending DeleteTablet for 3 replicas of tablet a74803512a3e4c7a80a219982143fbd7
I20260923 20:20:12.603241 23417 tablet_service.cc:1514] Processing DeleteTablet for tablet a74803512a3e4c7a80a219982143fbd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:12 UTC) from {username='slave'} at 127.0.0.1:41624
I20260923 20:20:12.603667 23547 tablet_service.cc:1514] Processing DeleteTablet for tablet a74803512a3e4c7a80a219982143fbd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:12 UTC) from {username='slave'} at 127.0.0.1:34162
I20260923 20:20:12.604467 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 23224
I20260923 20:20:12.604523 23286 tablet_service.cc:1514] Processing DeleteTablet for tablet a74803512a3e4c7a80a219982143fbd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:12 UTC) from {username='slave'} at 127.0.0.1:56638
I20260923 20:20:12.632082 23655 tablet_replica.cc:330] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791: stopping tablet replica
I20260923 20:20:12.633608 23656 tablet_replica.cc:330] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67: stopping tablet replica
I20260923 20:20:12.634135 23656 raft_consensus.cc:2229] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:20:12.634807 23656 raft_consensus.cc:2258] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:12.632758 23655 raft_consensus.cc:2229] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:20:12.635842 23655 raft_consensus.cc:2258] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:12.637748 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.605779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.5.83.65:46443)
0923 20:20:12.605968 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:12.605976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:12.628487 (+ 22511us) server_negotiation.cc:407] Connection header received
0923 20:20:12.628773 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:12.628779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:12.628910 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:12.629021 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:12.629698 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.630904 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.633149 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.633884 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.633988 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.635853 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:12.635864 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:12.635882 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:12.636801 (+ 919us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:12.636864 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:12.636891 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:12.636978 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:12.637240 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 20:20:12.637362 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:20:12.639086 23346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:12.641382 23655 ts_tablet_manager.cc:1905] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:12.645411 23654 tablet_replica.cc:330] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8: stopping tablet replica
I20260923 20:20:12.645910 23654 raft_consensus.cc:2229] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:20:12.646680 23654 raft_consensus.cc:2258] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:12.648699 23654 ts_tablet_manager.cc:1905] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:12.652413 23655 ts_tablet_manager.cc:1918] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:12.652619 23655 log.cc:1196] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/a74803512a3e4c7a80a219982143fbd7
I20260923 20:20:12.653175 23655 ts_tablet_manager.cc:1939] T a74803512a3e4c7a80a219982143fbd7 P 7d8b704293004b58b71aa472082e5791: Deleting consensus metadata
I20260923 20:20:12.655124 23151 catalog_manager.cc:4841] TS 7d8b704293004b58b71aa472082e5791 (127.5.83.67:45495): tablet a74803512a3e4c7a80a219982143fbd7 (table test-table [id=5fe53a5096d1493a865af81332e535b4]) successfully deleted
I20260923 20:20:12.655872 23654 ts_tablet_manager.cc:1918] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:12.656005 23654 log.cc:1196] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/a74803512a3e4c7a80a219982143fbd7
I20260923 20:20:12.656418 23654 ts_tablet_manager.cc:1939] T a74803512a3e4c7a80a219982143fbd7 P d24bab6a180e4d3ea178d76af4c11ec8: Deleting consensus metadata
I20260923 20:20:12.658012 23149 catalog_manager.cc:4841] TS d24bab6a180e4d3ea178d76af4c11ec8 (127.5.83.66:37929): tablet a74803512a3e4c7a80a219982143fbd7 (table test-table [id=5fe53a5096d1493a865af81332e535b4]) successfully deleted
I20260923 20:20:12.639104 23656 ts_tablet_manager.cc:1905] T a74803512a3e4c7a80a219982143fbd7 P 04d9ec181f3c49ca84599c93065f4e67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:12.758138 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 23224
I20260923 20:20:12.776621 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 23355
W20260923 20:20:12.776702 23151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.65:46443 (error 108)
I20260923 20:20:12.778748 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.778217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:46443 (local address 127.0.0.1:56648)
0923 20:20:12.778421 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.778570 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 20:20:12.779042 23151 catalog_manager.cc:4584] TS 04d9ec181f3c49ca84599c93065f4e67 (127.5.83.65:46443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a74803512a3e4c7a80a219982143fbd7: Network error: Client connection negotiation failed: client connection to 127.5.83.65:46443: connect: Connection refused (error 111)
I20260923 20:20:12.801689 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.777959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.5.83.66:37929)
0923 20:20:12.778152 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:12.778166 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:12.778186 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:20:12.778252 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:12.778258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:12.778365 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:12.778475 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:12.779092 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.780284 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.781842 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.782488 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.782625 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.799865 (+ 17240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:12.799879 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:12.799899 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:12.800795 (+ 896us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:12.800866 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:12.800896 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:12.801025 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:12.801166 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 20:20:12.801284 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:20:12.802480 23477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:12.832959 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.832402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:46443 (local address 127.0.0.1:56654)
0923 20:20:12.832664 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.832767 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260923 20:20:12.834239 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.833822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:46443 (local address 127.0.0.1:56658)
0923 20:20:12.833986 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.834071 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 20:20:12.894798 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.894235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:46443 (local address 127.0.0.1:56664)
0923 20:20:12.894487 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.894597 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 20:20:12.895951 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.895573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:46443 (local address 127.0.0.1:56676)
0923 20:20:12.895742 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.895816 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 20:20:12.924685 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 23355
I20260923 20:20:12.943710 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 23485
I20260923 20:20:12.950762 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.944515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.5.83.67:45495)
0923 20:20:12.944711 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:12.944717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:12.944829 (+ 112us) server_negotiation.cc:407] Connection header received
0923 20:20:12.945059 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:12.945063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:12.945149 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:12.945266 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:12.945658 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.946406 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.947744 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:12.948152 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:12.948292 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:12.949390 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:12.949399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:12.949415 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:12.949998 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:12.950050 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:12.950069 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:12.950204 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:12.950333 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 20:20:12.950455 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:20:12.951467 23607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:12.976498 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.975943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:46443 (local address 127.0.0.1:56678)
0923 20:20:12.976217 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.976320 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260923 20:20:12.977741 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:12.977341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:46443 (local address 127.0.0.1:56694)
0923 20:20:12.977555 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:12.977636 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260923 20:20:13.047981 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 23485
I20260923 20:20:13.066514 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 23134
I20260923 20:20:13.067680 23195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:13.181576 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 23134
I20260923 20:20:13.181649 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:13.180988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:46443 (local address 127.0.0.1:56702)
0923 20:20:13.181326 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:13.181411 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:46443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
2026-09-23T20:20:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2863 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:13 dist-test-slave-vl24 krb5kdc[23665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:13 dist-test-slave-vl24 krb5kdc[23665](info): set up 2 sockets
Sep 23 20:20:13 dist-test-slave-vl24 krb5kdc[23665](info): commencing operation
krb5kdc: starting...
W20260923 20:20:14.285988 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:14 dist-test-slave-vl24 krb5kdc[23665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:20:14.559787 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:33779
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:33779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:14.898116 23678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:14.898577 23678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:14.898749 23678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:14.906270 23678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:14.906445 23678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:14.906533 23678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:14.906617 23678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:14.906684 23678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:14.916857 23678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:33779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:33779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:14.918243 23678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:14.919553 23678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:14.931298 23683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:14.931319 23684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:14.931419 23686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:14.932566 23678 server_base.cc:1034] running on GCE node
I20260923 20:20:14.933458 23678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:14.934938 23678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:14.936201 23678 hybrid_clock.cc:648] HybridClock initialized: now 1790194814936159 us; error 60 us; skew 500 ppm
I20260923 20:20:14.936640 23678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:14 dist-test-slave-vl24 krb5kdc[23665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194814, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:14.939771 23678 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:20:14.941248 23678 webserver.cc:460] Webserver started at http://127.5.83.126:33723/ using document root <none> and password file <none>
I20260923 20:20:14.941892 23678 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:14.941994 23678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:14.942265 23678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:14.945348 23678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "b65b8abea19e470d901b32b1c70e5532"
format_stamp: "Formatted at 2026-09-23 20:20:14 on dist-test-slave-vl24"
server_key: "249dd306a9fc722449ea3efa8d70d58f"
server_key_iv: "a429ca7d4492e648b1a400d926e6ee7d"
server_key_version: "encryptionkey@0"
I20260923 20:20:14.946048 23678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "b65b8abea19e470d901b32b1c70e5532"
format_stamp: "Formatted at 2026-09-23 20:20:14 on dist-test-slave-vl24"
server_key: "249dd306a9fc722449ea3efa8d70d58f"
server_key_iv: "a429ca7d4492e648b1a400d926e6ee7d"
server_key_version: "encryptionkey@0"
I20260923 20:20:14.951043 23678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:20:14.954315 23693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:14.955752 23678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:20:14.955933 23678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "b65b8abea19e470d901b32b1c70e5532"
format_stamp: "Formatted at 2026-09-23 20:20:14 on dist-test-slave-vl24"
server_key: "249dd306a9fc722449ea3efa8d70d58f"
server_key_iv: "a429ca7d4492e648b1a400d926e6ee7d"
server_key_version: "encryptionkey@0"
I20260923 20:20:14.956122 23678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:14.978065 23678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:14.982249 23678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:14.982553 23678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:15.002589 23678 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:33779
I20260923 20:20:15.002627 23744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:33779 every 8 connection(s)
I20260923 20:20:15.003979 23678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:15.006628 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 23678
I20260923 20:20:15.006811 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:15.007356 5453 external_mini_cluster.cc:1428] Setting key 0eb7f92c83d6580e63c014d0a75affa5
I20260923 20:20:15.008323 23745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:20:15 dist-test-slave-vl24 krb5kdc[23665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:15.021001 23745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532: Bootstrap starting.
I20260923 20:20:15.024434 23745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:15.025522 23745 log.cc:824] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:15.028426 23745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532: No bootstrap required, opened a new log
I20260923 20:20:15.029651 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:15.009250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.5.83.126:33779)
0923 20:20:15.010039 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:15.010051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:15.010089 (+ 38us) server_negotiation.cc:407] Connection header received
0923 20:20:15.011654 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:15.011676 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:15.012046 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:15.012457 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:15.013956 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:15.015295 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:15.016689 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:15.017365 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:15.017601 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:15.022258 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:15.022268 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:15.022283 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:15.022325 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:15.025526 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:15.026264 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:15.026271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:15.026277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:15.026462 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:15.026873 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:15.026877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:15.026880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:15.027369 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:15.027726 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:15.028124 (+ 398us) server_negotiation.cc:299] Negotiation successful
0923 20:20:15.028641 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":367,"threads_started":1}
I20260923 20:20:15.036334 23745 raft_consensus.cc:348] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65b8abea19e470d901b32b1c70e5532" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 33779 } }
I20260923 20:20:15.036821 23745 raft_consensus.cc:374] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:15.036966 23745 raft_consensus.cc:729] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65b8abea19e470d901b32b1c70e5532, State: Initialized, Role: FOLLOWER
I20260923 20:20:15.037531 23745 consensus_queue.cc:260] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65b8abea19e470d901b32b1c70e5532" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 33779 } }
I20260923 20:20:15.037773 23745 raft_consensus.cc:388] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:15.037941 23745 raft_consensus.cc:482] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:15.038117 23745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:15.040292 23745 raft_consensus.cc:504] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65b8abea19e470d901b32b1c70e5532" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 33779 } }
I20260923 20:20:15.040961 23745 leader_election.cc:302] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b65b8abea19e470d901b32b1c70e5532; no voters:
I20260923 20:20:15.041633 23745 leader_election.cc:288] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:15.041800 23750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:15.042474 23750 raft_consensus.cc:686] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [term 1 LEADER]: Becoming Leader. State: Replica: b65b8abea19e470d901b32b1c70e5532, State: Running, Role: LEADER
I20260923 20:20:15.043025 23750 consensus_queue.cc:237] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65b8abea19e470d901b32b1c70e5532" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 33779 } }
I20260923 20:20:15.043800 23745 sys_catalog.cc:564] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:15.045959 23751 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b65b8abea19e470d901b32b1c70e5532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65b8abea19e470d901b32b1c70e5532" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 33779 } } }
I20260923 20:20:15.046334 23751 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:15.046756 23752 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b65b8abea19e470d901b32b1c70e5532. Latest consensus state: current_term: 1 leader_uuid: "b65b8abea19e470d901b32b1c70e5532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65b8abea19e470d901b32b1c70e5532" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 33779 } } }
I20260923 20:20:15.047084 23752 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:15.048050 23757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:15.053927 23757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:15.063200 23757 catalog_manager.cc:1347] Generated new cluster ID: e256a601d1494662af94ea940edcf087
I20260923 20:20:15.063324 23757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:15.076908 23757 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:15.078817 23757 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:15.094077 23757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532: Generated new TSK 0
I20260923 20:20:15.095309 23757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:20:15.167078 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:33779
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:15.532425 23771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:15.532796 23771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:15.532895 23771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:15.540385 23771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:15.540503 23771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:15.540676 23771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:20:15.551177 23771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:33779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:15.552475 23771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:15.553835 23771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:15.567319 23776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:15.567268 23777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:15.567241 23779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:15.568056 23771 server_base.cc:1034] running on GCE node
I20260923 20:20:15.568735 23771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:15.569707 23771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:15.570915 23771 hybrid_clock.cc:648] HybridClock initialized: now 1790194815570871 us; error 42 us; skew 500 ppm
I20260923 20:20:15.571231 23771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:15 dist-test-slave-vl24 krb5kdc[23665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194815, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:15.574554 23771 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:20:15.575958 23771 webserver.cc:460] Webserver started at http://127.5.83.65:33681/ using document root <none> and password file <none>
I20260923 20:20:15.576689 23771 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:15.576794 23771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:15.577057 23771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:15.579944 23771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "878006e777a0407499425b5a8cf88301"
format_stamp: "Formatted at 2026-09-23 20:20:15 on dist-test-slave-vl24"
server_key: "05da0e37dab0281e6f1627fee7209e01"
server_key_iv: "fe6432519ee084057ddff2691811ba4c"
server_key_version: "encryptionkey@0"
I20260923 20:20:15.580778 23771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "878006e777a0407499425b5a8cf88301"
format_stamp: "Formatted at 2026-09-23 20:20:15 on dist-test-slave-vl24"
server_key: "05da0e37dab0281e6f1627fee7209e01"
server_key_iv: "fe6432519ee084057ddff2691811ba4c"
server_key_version: "encryptionkey@0"
I20260923 20:20:15.586184 23771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:20:15.589628 23786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:15.590993 23771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:20:15.591166 23771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "878006e777a0407499425b5a8cf88301"
format_stamp: "Formatted at 2026-09-23 20:20:15 on dist-test-slave-vl24"
server_key: "05da0e37dab0281e6f1627fee7209e01"
server_key_iv: "fe6432519ee084057ddff2691811ba4c"
server_key_version: "encryptionkey@0"
I20260923 20:20:15.591357 23771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:15.606190 23771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:15.610807 23771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:15.611124 23771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:15.612151 23771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:15.613577 23771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:15.613653 23771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:15.613823 23771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:15.613905 23771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:15.649101 23771 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:38299
I20260923 20:20:15.649158 23898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:38299 every 8 connection(s)
I20260923 20:20:15.650606 23771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:20:15.656344 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 23771
I20260923 20:20:15.656574 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:20:15.657130 5453 external_mini_cluster.cc:1428] Setting key 2ff0241df09a0234453c0dd4cd0ab42b
Sep 23 20:20:15 dist-test-slave-vl24 krb5kdc[23665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194815, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:15.671006 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:15.653358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:51127 (local address 127.5.83.126:33779)
0923 20:20:15.653802 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:15.653807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:15.654779 (+ 972us) server_negotiation.cc:407] Connection header received
0923 20:20:15.656670 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:15.656676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:15.656766 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:15.656889 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:15.658472 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:15.659315 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:15.661013 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:15.661413 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:15.661555 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:15.665294 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:15.665300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:15.665303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:15.665335 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:15.667831 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:15.668663 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:15.668676 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:15.668680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:15.668805 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:15.669249 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:15.669253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:15.669255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:15.669519 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:15.669673 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:15.670398 (+ 725us) server_negotiation.cc:299] Negotiation successful
0923 20:20:15.670625 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":220,"threads_started":1}
I20260923 20:20:15.672001 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:15.653896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:33779 (local address 127.5.83.65:51127)
0923 20:20:15.654615 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:15.654655 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:15.656293 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:15.657072 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:15.657079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:15.657622 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:15.658303 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:15.658322 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:15.659485 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:15.659490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:15.660875 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:15.660888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:15.661645 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:15.661660 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:15.661901 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:15.662603 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:20:15.662626 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:15.665055 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:15.668040 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:15.668050 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:15.668067 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:15.668508 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:15.668918 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:15.668922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:15.668928 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:15.669124 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:15.669657 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:15.669665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:15.670014 (+ 349us) client_negotiation.cc:769] Sending connection context
0923 20:20:15.670281 (+ 267us) client_negotiation.cc:240] Negotiation successful
0923 20:20:15.670840 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":287,"threads_started":1}
I20260923 20:20:15.673655 23899 heartbeater.cc:347] Connected to a master server at 127.5.83.126:33779
I20260923 20:20:15.673969 23899 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:15.674669 23899 heartbeater.cc:511] Master 127.5.83.126:33779 requested a full tablet report, sending...
I20260923 20:20:15.677006 23710 ts_manager.cc:194] Registered new tserver with Master: 878006e777a0407499425b5a8cf88301 (127.5.83.65:38299)
I20260923 20:20:15.679229 23710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:51127
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:20:15.719652 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:33779
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:16.064486 23905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:16.064833 23905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:16.065016 23905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:16.073714 23905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:16.073939 23905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:16.074156 23905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:20:16.085186 23905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:33779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:16.086652 23905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:16.088150 23905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:16.102232 23910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:16.102415 23911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:16.102368 23913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:16.103581 23905 server_base.cc:1034] running on GCE node
I20260923 20:20:16.104204 23905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:16.105223 23905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:16.106456 23905 hybrid_clock.cc:648] HybridClock initialized: now 1790194816106399 us; error 40 us; skew 500 ppm
I20260923 20:20:16.106783 23905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:16 dist-test-slave-vl24 krb5kdc[23665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194816, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:16.110142 23905 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:20:16.111636 23905 webserver.cc:460] Webserver started at http://127.5.83.66:39677/ using document root <none> and password file <none>
I20260923 20:20:16.112325 23905 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:16.112473 23905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:16.112802 23905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:16.115643 23905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "e77c602949b242fca02bf41b45a34193"
format_stamp: "Formatted at 2026-09-23 20:20:16 on dist-test-slave-vl24"
server_key: "a38fb8cfaf015562fd2aba995582e532"
server_key_iv: "f9ff532be53adb0518405335f4c98ff5"
server_key_version: "encryptionkey@0"
I20260923 20:20:16.116446 23905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "e77c602949b242fca02bf41b45a34193"
format_stamp: "Formatted at 2026-09-23 20:20:16 on dist-test-slave-vl24"
server_key: "a38fb8cfaf015562fd2aba995582e532"
server_key_iv: "f9ff532be53adb0518405335f4c98ff5"
server_key_version: "encryptionkey@0"
I20260923 20:20:16.121657 23905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:20:16.124938 23920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:16.126241 23905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:20:16.126408 23905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "e77c602949b242fca02bf41b45a34193"
format_stamp: "Formatted at 2026-09-23 20:20:16 on dist-test-slave-vl24"
server_key: "a38fb8cfaf015562fd2aba995582e532"
server_key_iv: "f9ff532be53adb0518405335f4c98ff5"
server_key_version: "encryptionkey@0"
I20260923 20:20:16.126623 23905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:16.156169 23905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:16.161063 23905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:16.161366 23905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:16.162398 23905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:16.163769 23905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:16.163842 23905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:16.163995 23905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:16.164043 23905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:16.198392 23905 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:35797
I20260923 20:20:16.198494 24032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:35797 every 8 connection(s)
I20260923 20:20:16.199862 23905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:20:16.209501 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 23905
I20260923 20:20:16.209688 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:20:16.210247 5453 external_mini_cluster.cc:1428] Setting key 89a592e5852b7f48d70090b37fa8cf18
Sep 23 20:20:16 dist-test-slave-vl24 krb5kdc[23665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194816, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:16.222535 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.202542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:54895 (local address 127.5.83.126:33779)
0923 20:20:16.202980 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:16.202985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:16.203811 (+ 826us) server_negotiation.cc:407] Connection header received
0923 20:20:16.205760 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:16.205766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:16.205861 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:16.205988 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:16.207492 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.208209 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.210394 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.211013 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.211171 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.215478 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:16.215485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:16.215488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:16.215527 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:16.218540 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:16.219393 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:16.219399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:16.219401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:16.219545 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:16.220043 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:16.220055 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:16.220057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:16.220354 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:16.220516 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:16.221793 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0923 20:20:16.222082 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":232,"threads_started":1}
I20260923 20:20:16.223122 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.202959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:33779 (local address 127.5.83.66:54895)
0923 20:20:16.203651 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:16.203688 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:16.205493 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:16.206229 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:16.206238 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:16.206733 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:16.207300 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.207318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.208381 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.208386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.210159 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.210175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.211265 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.211272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.211542 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.212351 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:20:16.212402 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:16.215194 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:16.218714 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:16.218724 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:16.218744 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:16.219239 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:16.219671 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:16.219677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:16.219681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:16.219904 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:16.220506 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:16.220514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:16.221044 (+ 530us) client_negotiation.cc:769] Sending connection context
0923 20:20:16.221358 (+ 314us) client_negotiation.cc:240] Negotiation successful
0923 20:20:16.221891 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":294,"threads_started":1}
I20260923 20:20:16.224737 24033 heartbeater.cc:347] Connected to a master server at 127.5.83.126:33779
I20260923 20:20:16.225090 24033 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:16.225978 24033 heartbeater.cc:511] Master 127.5.83.126:33779 requested a full tablet report, sending...
I20260923 20:20:16.227578 23710 ts_manager.cc:194] Registered new tserver with Master: e77c602949b242fca02bf41b45a34193 (127.5.83.66:35797)
I20260923 20:20:16.228410 23710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:54895
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:20:16.272248 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:33779
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:40793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:16.607075 24039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:16.607403 24039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:16.607537 24039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:16.615202 24039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:16.615329 24039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:16.615444 24039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:20:16.625625 24039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:40793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:33779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:16.626874 24039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:16.628155 24039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:16.641551 24044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:16.641621 24047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:16.641678 24045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:16.643119 24039 server_base.cc:1034] running on GCE node
I20260923 20:20:16.643791 24039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:16.644835 24039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:16.646049 24039 hybrid_clock.cc:648] HybridClock initialized: now 1790194816645974 us; error 62 us; skew 500 ppm
I20260923 20:20:16.646392 24039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:16 dist-test-slave-vl24 krb5kdc[23665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194816, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:16.649856 24039 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:20:16.651579 24039 webserver.cc:460] Webserver started at http://127.5.83.67:36289/ using document root <none> and password file <none>
I20260923 20:20:16.652288 24039 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:16.652415 24039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:16.652786 24039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:16.655711 24039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "8513558cf9c64f0a8449077cfd798158"
format_stamp: "Formatted at 2026-09-23 20:20:16 on dist-test-slave-vl24"
server_key: "2c7593398cd9186be20c5fcdcc8f4ee8"
server_key_iv: "8910e61efe1f119ed118f9e1aa00d8ae"
server_key_version: "encryptionkey@0"
I20260923 20:20:16.656512 24039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "8513558cf9c64f0a8449077cfd798158"
format_stamp: "Formatted at 2026-09-23 20:20:16 on dist-test-slave-vl24"
server_key: "2c7593398cd9186be20c5fcdcc8f4ee8"
server_key_iv: "8910e61efe1f119ed118f9e1aa00d8ae"
server_key_version: "encryptionkey@0"
I20260923 20:20:16.661865 24039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:20:16.665252 24054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:16.666508 24039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:20:16.666678 24039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "8513558cf9c64f0a8449077cfd798158"
format_stamp: "Formatted at 2026-09-23 20:20:16 on dist-test-slave-vl24"
server_key: "2c7593398cd9186be20c5fcdcc8f4ee8"
server_key_iv: "8910e61efe1f119ed118f9e1aa00d8ae"
server_key_version: "encryptionkey@0"
I20260923 20:20:16.666882 24039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:16.680719 24039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:16.682323 23899 heartbeater.cc:503] Master 127.5.83.126:33779 was elected leader, sending a full tablet report...
I20260923 20:20:16.685091 24039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:16.685374 24039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:16.686297 24039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:16.687525 24039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:16.687623 24039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:16.687773 24039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:16.687847 24039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:16.721169 24039 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:42239
I20260923 20:20:16.721216 24166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:42239 every 8 connection(s)
I20260923 20:20:16.722517 24039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:20:16.730162 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 24039
I20260923 20:20:16.730340 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:20:16.730849 5453 external_mini_cluster.cc:1428] Setting key 065fb913a6f33241c82675e7e6a564c2
Sep 23 20:20:16 dist-test-slave-vl24 krb5kdc[23665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194816, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:16.742229 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.725007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:46317 (local address 127.5.83.126:33779)
0923 20:20:16.725468 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:16.725473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:16.726251 (+ 778us) server_negotiation.cc:407] Connection header received
0923 20:20:16.728054 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:16.728060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:16.728149 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:16.728266 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:16.729845 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.730829 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.732576 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.732978 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.733111 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.736587 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:16.736593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:16.736597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:16.736632 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:16.739329 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:16.740107 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:16.740111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:16.740114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:16.740230 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:16.740685 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:16.740688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:16.740690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:16.740961 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:16.741073 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:16.741780 (+ 707us) server_negotiation.cc:299] Negotiation successful
0923 20:20:16.741959 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260923 20:20:16.743338 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.725384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:33779 (local address 127.5.83.67:46317)
0923 20:20:16.726101 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:16.726137 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:16.727782 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:16.728494 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:16.728501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:16.729046 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:16.729658 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.729677 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.730973 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.730979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.732398 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.732410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.733216 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.733226 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.733459 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.734149 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:20:16.734172 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:16.736314 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:16.739504 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:16.739512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:16.739526 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:16.739998 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:16.740325 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:16.740328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:16.740331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:16.740515 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:16.741090 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:16.741096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:16.741427 (+ 331us) client_negotiation.cc:769] Sending connection context
0923 20:20:16.741694 (+ 267us) client_negotiation.cc:240] Negotiation successful
0923 20:20:16.742092 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":301,"threads_started":1}
I20260923 20:20:16.744884 24167 heartbeater.cc:347] Connected to a master server at 127.5.83.126:33779
I20260923 20:20:16.745247 24167 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:16.746210 24167 heartbeater.cc:511] Master 127.5.83.126:33779 requested a full tablet report, sending...
I20260923 20:20:16.747650 23710 ts_manager.cc:194] Registered new tserver with Master: 8513558cf9c64f0a8449077cfd798158 (127.5.83.67:42239)
I20260923 20:20:16.748469 23710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:46317
I20260923 20:20:16.758642 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:20:16.776078 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.762694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.5.83.126:33779)
0923 20:20:16.762945 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:16.762955 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:16.763353 (+ 398us) server_negotiation.cc:407] Connection header received
0923 20:20:16.763493 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:16.763499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:16.763586 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:16.763675 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:16.765141 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.765869 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.767093 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.767471 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.767564 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.770386 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:16.770392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:16.770395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:16.770433 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:16.773442 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:16.774231 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:16.774237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:16.774239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:16.774369 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:16.774766 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:16.774770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:16.774772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:16.775108 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:16.775334 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:16.775662 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 20:20:16.775837 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:20:16.780318 23710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:20:16.787302 23710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:20:16.811551 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.799725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:35797 (local address 127.0.0.1:44092)
0923 20:20:16.800619 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:16.800667 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:16.800862 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:16.802771 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:16.802782 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:16.802814 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:16.803388 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.803401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.805672 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.805677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.808769 (+ 3092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.808785 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.810531 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.810539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.810720 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.810783 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 20:20:16.810887 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 20:20:16.811076 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":8,"thread_start_us":380,"threads_started":1}
I20260923 20:20:16.812088 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.800133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.5.83.66:35797)
0923 20:20:16.801895 (+ 1762us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:16.801902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:16.801931 (+ 29us) server_negotiation.cc:407] Connection header received
0923 20:20:16.802043 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:16.802051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:16.802315 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:16.803520 (+ 1205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:16.803751 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.805507 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.808932 (+ 3425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.810291 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.811305 (+ 1014us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.811510 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:16.811589 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 20:20:16.811770 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1594,"thread_start_us":1496,"threads_started":1}
I20260923 20:20:16.813020 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.799764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.5.83.67:42239)
0923 20:20:16.801634 (+ 1870us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:16.801645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:16.802333 (+ 688us) server_negotiation.cc:407] Connection header received
0923 20:20:16.802435 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:16.802442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:16.802666 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:16.802833 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:16.804146 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.805680 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.810040 (+ 4360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.811495 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.811609 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.811821 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:16.812485 (+ 664us) server_negotiation.cc:299] Negotiation successful
0923 20:20:16.812687 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1688,"thread_start_us":1544,"threads_started":1}
I20260923 20:20:16.813573 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.799729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:42239 (local address 127.0.0.1:40194)
0923 20:20:16.802064 (+ 2335us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:16.802080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:16.802193 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:16.803100 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:16.803105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:16.803133 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:16.803913 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.803925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.805831 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.805838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.809870 (+ 4032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.809882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.811985 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.811996 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.812231 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.812275 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 20:20:16.812372 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:20:16.813318 (+ 946us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2199,"mutex_wait_us":2720,"thread_start_us":1846,"threads_started":1}
I20260923 20:20:16.816094 23968 tablet_service.cc:1467] Processing CreateTablet for tablet 01ae35c553b94691be675b688315dcb1 (DEFAULT_TABLE table=test-table [id=f34ff0b13f2645339ee31e1aad4e7c00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:16.816805 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.802141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:38299 (local address 127.0.0.1:55240)
0923 20:20:16.808609 (+ 6468us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:16.808640 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:16.808789 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:16.809528 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:16.809532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:16.809559 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:16.810011 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.810025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.811902 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.811907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.814471 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.814488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.816062 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.816066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.816180 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.816200 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 20:20:16.816306 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:20:16.816430 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6315,"thread_start_us":6198,"threads_started":1}
I20260923 20:20:16.816870 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.802540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.5.83.65:38299)
0923 20:20:16.808199 (+ 5659us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:16.808205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:16.808749 (+ 544us) server_negotiation.cc:407] Connection header received
0923 20:20:16.809074 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:16.809083 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:16.809312 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:16.809539 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:16.810199 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.811744 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.814675 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.815672 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.815751 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.815907 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:16.816353 (+ 446us) server_negotiation.cc:299] Negotiation successful
0923 20:20:16.816562 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5474,"thread_start_us":5307,"threads_started":1}
I20260923 20:20:16.817710 23968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ae35c553b94691be675b688315dcb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:16.817248 24102 tablet_service.cc:1467] Processing CreateTablet for tablet 01ae35c553b94691be675b688315dcb1 (DEFAULT_TABLE table=test-table [id=f34ff0b13f2645339ee31e1aad4e7c00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:16.818638 24102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ae35c553b94691be675b688315dcb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:16.819892 23834 tablet_service.cc:1467] Processing CreateTablet for tablet 01ae35c553b94691be675b688315dcb1 (DEFAULT_TABLE table=test-table [id=f34ff0b13f2645339ee31e1aad4e7c00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:16.821111 23834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ae35c553b94691be675b688315dcb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:16.832474 24186 tablet_bootstrap.cc:492] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301: Bootstrap starting.
I20260923 20:20:16.834712 24188 tablet_bootstrap.cc:492] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158: Bootstrap starting.
I20260923 20:20:16.837693 24187 tablet_bootstrap.cc:492] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193: Bootstrap starting.
I20260923 20:20:16.838234 24186 tablet_bootstrap.cc:654] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:16.838439 24188 tablet_bootstrap.cc:654] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:16.839680 24186 log.cc:824] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:16.839680 24188 log.cc:824] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:16.840730 24187 tablet_bootstrap.cc:654] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:16.842024 24187 log.cc:824] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:16.842924 24186 tablet_bootstrap.cc:492] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301: No bootstrap required, opened a new log
I20260923 20:20:16.843145 24188 tablet_bootstrap.cc:492] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158: No bootstrap required, opened a new log
I20260923 20:20:16.843221 24186 ts_tablet_manager.cc:1397] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 20:20:16.843444 24188 ts_tablet_manager.cc:1397] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260923 20:20:16.846536 24187 tablet_bootstrap.cc:492] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193: No bootstrap required, opened a new log
I20260923 20:20:16.846788 24187 ts_tablet_manager.cc:1397] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260923 20:20:16.853775 24188 raft_consensus.cc:348] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8513558cf9c64f0a8449077cfd798158" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42239 } } peers { permanent_uuid: "e77c602949b242fca02bf41b45a34193" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35797 } } peers { permanent_uuid: "878006e777a0407499425b5a8cf88301" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 38299 } }
I20260923 20:20:16.854187 24188 raft_consensus.cc:374] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:16.854291 24188 raft_consensus.cc:729] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8513558cf9c64f0a8449077cfd798158, State: Initialized, Role: FOLLOWER
I20260923 20:20:16.854099 24186 raft_consensus.cc:348] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8513558cf9c64f0a8449077cfd798158" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42239 } } peers { permanent_uuid: "e77c602949b242fca02bf41b45a34193" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35797 } } peers { permanent_uuid: "878006e777a0407499425b5a8cf88301" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 38299 } }
I20260923 20:20:16.854530 24186 raft_consensus.cc:374] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:16.854621 24186 raft_consensus.cc:729] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878006e777a0407499425b5a8cf88301, State: Initialized, Role: FOLLOWER
I20260923 20:20:16.855007 24188 consensus_queue.cc:260] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8513558cf9c64f0a8449077cfd798158" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42239 } } peers { permanent_uuid: "e77c602949b242fca02bf41b45a34193" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35797 } } peers { permanent_uuid: "878006e777a0407499425b5a8cf88301" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 38299 } }
I20260923 20:20:16.855185 24186 consensus_queue.cc:260] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8513558cf9c64f0a8449077cfd798158" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42239 } } peers { permanent_uuid: "e77c602949b242fca02bf41b45a34193" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35797 } } peers { permanent_uuid: "878006e777a0407499425b5a8cf88301" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 38299 } }
I20260923 20:20:16.858266 24187 raft_consensus.cc:348] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8513558cf9c64f0a8449077cfd798158" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42239 } } peers { permanent_uuid: "e77c602949b242fca02bf41b45a34193" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35797 } } peers { permanent_uuid: "878006e777a0407499425b5a8cf88301" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 38299 } }
I20260923 20:20:16.858713 24187 raft_consensus.cc:374] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:16.858814 24187 raft_consensus.cc:729] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77c602949b242fca02bf41b45a34193, State: Initialized, Role: FOLLOWER
I20260923 20:20:16.858878 24188 ts_tablet_manager.cc:1428] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260923 20:20:16.859243 24186 ts_tablet_manager.cc:1428] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260923 20:20:16.859427 24187 consensus_queue.cc:260] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8513558cf9c64f0a8449077cfd798158" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42239 } } peers { permanent_uuid: "e77c602949b242fca02bf41b45a34193" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35797 } } peers { permanent_uuid: "878006e777a0407499425b5a8cf88301" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 38299 } }
I20260923 20:20:16.860258 24167 heartbeater.cc:503] Master 127.5.83.126:33779 was elected leader, sending a full tablet report...
I20260923 20:20:16.866456 24033 heartbeater.cc:503] Master 127.5.83.126:33779 was elected leader, sending a full tablet report...
I20260923 20:20:16.867205 24187 ts_tablet_manager.cc:1428] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193: Time spent starting tablet: real 0.020s user 0.011s sys 0.009s
W20260923 20:20:16.903115 23900 tablet.cc:2369] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:20:16.915573 24192 raft_consensus.cc:482] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:20:16.915855 24192 raft_consensus.cc:504] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8513558cf9c64f0a8449077cfd798158" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42239 } } peers { permanent_uuid: "e77c602949b242fca02bf41b45a34193" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35797 } } peers { permanent_uuid: "878006e777a0407499425b5a8cf88301" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 38299 } }
I20260923 20:20:16.917435 24192 leader_election.cc:288] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e77c602949b242fca02bf41b45a34193 (127.5.83.66:35797), 878006e777a0407499425b5a8cf88301 (127.5.83.65:38299)
I20260923 20:20:16.922927 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.917986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:37207 (local address 127.5.83.65:38299)
0923 20:20:16.918182 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:16.918190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:16.918211 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:20:16.918289 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:16.918298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:16.918386 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:16.918493 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:16.918921 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.919630 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.921275 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.922050 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.922197 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.922297 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:16.922554 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 20:20:16.922679 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 20:20:16.922945 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.917731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:38299 (local address 127.5.83.67:37207)
0923 20:20:16.917965 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:16.917985 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:16.918121 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:16.918524 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:16.918532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:16.918572 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:16.918755 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.918764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.919755 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.919763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.921119 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.921130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.922209 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.922214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.922397 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.922421 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:20:16.922527 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 20:20:16.922654 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":19}
I20260923 20:20:16.923926 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.917818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:35797 (local address 127.5.83.67:54441)
0923 20:20:16.918681 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:16.918694 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:16.918831 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:16.919161 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:16.919165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:16.919186 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:16.919500 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.919508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.920480 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.920484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.922011 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:16.922023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.923272 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.923280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:16.923467 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.923498 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 20:20:16.923595 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 20:20:16.923676 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":18,"thread_start_us":543,"threads_started":1}
I20260923 20:20:16.924034 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.918195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:54441 (local address 127.5.83.66:35797)
0923 20:20:16.918422 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:16.918428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:16.918786 (+ 358us) server_negotiation.cc:407] Connection header received
0923 20:20:16.918981 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:16.918987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:16.919048 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:16.919146 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:16.919642 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.920322 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.922152 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.922934 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.923097 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.923197 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:16.923646 (+ 449us) server_negotiation.cc:299] Negotiation successful
0923 20:20:16.923784 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:20:16.924077 23854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ae35c553b94691be675b688315dcb1" candidate_uuid: "8513558cf9c64f0a8449077cfd798158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878006e777a0407499425b5a8cf88301" is_pre_election: true
I20260923 20:20:16.924701 23854 raft_consensus.cc:2454] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8513558cf9c64f0a8449077cfd798158 in term 0.
I20260923 20:20:16.924990 23988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ae35c553b94691be675b688315dcb1" candidate_uuid: "8513558cf9c64f0a8449077cfd798158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77c602949b242fca02bf41b45a34193" is_pre_election: true
I20260923 20:20:16.925376 24055 leader_election.cc:302] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8513558cf9c64f0a8449077cfd798158, 878006e777a0407499425b5a8cf88301; no voters:
I20260923 20:20:16.925527 23988 raft_consensus.cc:2454] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8513558cf9c64f0a8449077cfd798158 in term 0.
I20260923 20:20:16.925738 24192 raft_consensus.cc:2781] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:20:16.925876 24192 raft_consensus.cc:482] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:20:16.926009 24192 raft_consensus.cc:3037] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:16.928192 24192 raft_consensus.cc:504] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8513558cf9c64f0a8449077cfd798158" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42239 } } peers { permanent_uuid: "e77c602949b242fca02bf41b45a34193" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35797 } } peers { permanent_uuid: "878006e777a0407499425b5a8cf88301" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 38299 } }
I20260923 20:20:16.928917 24192 leader_election.cc:288] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [CANDIDATE]: Term 1 election: Requested vote from peers e77c602949b242fca02bf41b45a34193 (127.5.83.66:35797), 878006e777a0407499425b5a8cf88301 (127.5.83.65:38299)
I20260923 20:20:16.929426 23988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ae35c553b94691be675b688315dcb1" candidate_uuid: "8513558cf9c64f0a8449077cfd798158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77c602949b242fca02bf41b45a34193"
I20260923 20:20:16.929656 23988 raft_consensus.cc:3037] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:16.930086 23854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ae35c553b94691be675b688315dcb1" candidate_uuid: "8513558cf9c64f0a8449077cfd798158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878006e777a0407499425b5a8cf88301"
I20260923 20:20:16.930313 23854 raft_consensus.cc:3037] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:16.932372 23988 raft_consensus.cc:2454] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8513558cf9c64f0a8449077cfd798158 in term 1.
I20260923 20:20:16.932719 23854 raft_consensus.cc:2454] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8513558cf9c64f0a8449077cfd798158 in term 1.
I20260923 20:20:16.932781 24057 leader_election.cc:302] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8513558cf9c64f0a8449077cfd798158, e77c602949b242fca02bf41b45a34193; no voters:
I20260923 20:20:16.933215 24192 raft_consensus.cc:2781] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:16.933637 24192 raft_consensus.cc:686] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 1 LEADER]: Becoming Leader. State: Replica: 8513558cf9c64f0a8449077cfd798158, State: Running, Role: LEADER
I20260923 20:20:16.934095 24192 consensus_queue.cc:237] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8513558cf9c64f0a8449077cfd798158" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42239 } } peers { permanent_uuid: "e77c602949b242fca02bf41b45a34193" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35797 } } peers { permanent_uuid: "878006e777a0407499425b5a8cf88301" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 38299 } }
I20260923 20:20:16.938414 23710 catalog_manager.cc:5495] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8513558cf9c64f0a8449077cfd798158 (127.5.83.67). New cstate: current_term: 1 leader_uuid: "8513558cf9c64f0a8449077cfd798158" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8513558cf9c64f0a8449077cfd798158" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 42239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77c602949b242fca02bf41b45a34193" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878006e777a0407499425b5a8cf88301" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 38299 } health_report { overall_health: UNKNOWN } } }
W20260923 20:20:16.952028 24034 tablet.cc:2369] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:20:16.973536 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:16.967667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40204 (local address 127.5.83.67:42239)
0923 20:20:16.967851 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:16.967856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:16.967875 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:20:16.968068 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:16.968072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:16.968160 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:16.968290 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:20:16.968713 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.969526 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.971017 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:16.971448 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:16.971599 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:16.972233 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:20:16.972922 (+ 689us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:20:16.973074 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:16.973262 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 20:20:16.973307 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 20:20:16.974233 24168 tablet.cc:2369] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:20:16.982225 23854 raft_consensus.cc:1261] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301 [term 1 FOLLOWER]: Refusing update from remote peer 8513558cf9c64f0a8449077cfd798158: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:20:16.982218 23988 raft_consensus.cc:1261] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193 [term 1 FOLLOWER]: Refusing update from remote peer 8513558cf9c64f0a8449077cfd798158: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:20:16.983235 24196 consensus_queue.cc:1035] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77c602949b242fca02bf41b45a34193" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:20:16.983592 24192 consensus_queue.cc:1035] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "878006e777a0407499425b5a8cf88301" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:20:16.996883 24199 mvcc.cc:205] Tried to move back new op lower bound from 7332637970348728320 to 7332637970165637120. Current Snapshot: MvccSnapshot[applied={T|T < 7332637970348728320}]
I20260923 20:20:16.997155 24200 mvcc.cc:205] Tried to move back new op lower bound from 7332637970348728320 to 7332637970165637120. Current Snapshot: MvccSnapshot[applied={T|T < 7332637970348728320}]
I20260923 20:20:16.997496 24201 mvcc.cc:205] Tried to move back new op lower bound from 7332637970348728320 to 7332637970165637120. Current Snapshot: MvccSnapshot[applied={T|T < 7332637970348728320}]
I20260923 20:20:17.012477 23710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:20:17.012820 23710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:20:17.017534 23710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b65b8abea19e470d901b32b1c70e5532: Sending DeleteTablet for 3 replicas of tablet 01ae35c553b94691be675b688315dcb1
I20260923 20:20:17.018970 23834 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ae35c553b94691be675b688315dcb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:17 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:55240
I20260923 20:20:17.018958 23968 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ae35c553b94691be675b688315dcb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:17 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:44092
I20260923 20:20:17.019184 24102 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ae35c553b94691be675b688315dcb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:17 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:40194
I20260923 20:20:17.022840 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 23771
I20260923 20:20:17.051009 24209 tablet_replica.cc:330] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193: stopping tablet replica
I20260923 20:20:17.051649 24211 tablet_replica.cc:330] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158: stopping tablet replica
I20260923 20:20:17.051584 24209 raft_consensus.cc:2229] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:20:17.052178 24209 raft_consensus.cc:2258] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:17.052701 24210 tablet_replica.cc:330] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301: stopping tablet replica
I20260923 20:20:17.052175 24211 raft_consensus.cc:2229] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:20:17.053107 24210 raft_consensus.cc:2229] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:20:17.053611 24210 raft_consensus.cc:2258] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:17.054057 24211 raft_consensus.cc:2258] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:17.056389 24210 ts_tablet_manager.cc:1905] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:17.057142 24209 ts_tablet_manager.cc:1905] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:17.057749 24211 ts_tablet_manager.cc:1905] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 20:20:17 dist-test-slave-vl24 krb5kdc[23665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:20:17.064401 24209 ts_tablet_manager.cc:1918] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:17.064599 24209 log.cc:1196] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/01ae35c553b94691be675b688315dcb1
I20260923 20:20:17.065168 24209 ts_tablet_manager.cc:1939] T 01ae35c553b94691be675b688315dcb1 P e77c602949b242fca02bf41b45a34193: Deleting consensus metadata
I20260923 20:20:17.067153 23696 catalog_manager.cc:4841] TS e77c602949b242fca02bf41b45a34193 (127.5.83.66:35797): tablet 01ae35c553b94691be675b688315dcb1 (table test-table [id=f34ff0b13f2645339ee31e1aad4e7c00]) successfully deleted
I20260923 20:20:17.067987 24211 ts_tablet_manager.cc:1918] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:17.068169 24211 log.cc:1196] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/01ae35c553b94691be675b688315dcb1
I20260923 20:20:17.068763 24211 ts_tablet_manager.cc:1939] T 01ae35c553b94691be675b688315dcb1 P 8513558cf9c64f0a8449077cfd798158: Deleting consensus metadata
I20260923 20:20:17.070969 23694 catalog_manager.cc:4841] TS 8513558cf9c64f0a8449077cfd798158 (127.5.83.67:42239): tablet 01ae35c553b94691be675b688315dcb1 (table test-table [id=f34ff0b13f2645339ee31e1aad4e7c00]) successfully deleted
I20260923 20:20:17.072702 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:17.024353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.5.83.65:38299)
0923 20:20:17.024626 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:17.024635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:17.042465 (+ 17830us) server_negotiation.cc:407] Connection header received
0923 20:20:17.043110 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:17.043121 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:17.043291 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:17.043398 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:17.046197 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:17.047771 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:17.054725 (+ 6954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:17.055453 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:17.055589 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:17.062880 (+ 7291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:17.062902 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:17.062926 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:17.064896 (+ 1970us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:17.068906 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:17.069891 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:17.069901 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:17.069905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:17.070087 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:17.070556 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:17.070562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:17.070565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:17.071038 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:17.071321 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:17.071983 (+ 662us) server_negotiation.cc:299] Negotiation successful
0923 20:20:17.072238 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:20:17.074230 23894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:17.247354 24210 ts_tablet_manager.cc:1918] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:17.247594 24210 log.cc:1196] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790194506409421-5453-0/minicluster-data/ts-0/wal/wals/01ae35c553b94691be675b688315dcb1
I20260923 20:20:17.247766 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 23771
I20260923 20:20:17.248185 24210 ts_tablet_manager.cc:1939] T 01ae35c553b94691be675b688315dcb1 P 878006e777a0407499425b5a8cf88301: Deleting consensus metadata
W20260923 20:20:17.277086 23694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.65:38299 (error 108)
I20260923 20:20:17.277333 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 23905
I20260923 20:20:17.290323 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:17.289915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:38299 (local address 127.0.0.1:55254)
0923 20:20:17.290072 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:17.290187 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 20:20:17.290729 23694 catalog_manager.cc:4584] TS 878006e777a0407499425b5a8cf88301 (127.5.83.65:38299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01ae35c553b94691be675b688315dcb1: Network error: Client connection negotiation failed: client connection to 127.5.83.65:38299: connect: Connection refused (error 111)
Sep 23 20:20:17 dist-test-slave-vl24 krb5kdc[23665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:20:17.301487 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:17.279065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.5.83.66:35797)
0923 20:20:17.279276 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:17.279284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:17.279306 (+ 22us) server_negotiation.cc:407] Connection header received
0923 20:20:17.279377 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:17.279385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:17.279502 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:17.279610 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:17.281741 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:17.283065 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:17.285373 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:17.286146 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:17.286269 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:17.292704 (+ 6435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:17.292724 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:17.292743 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:17.292793 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:17.297933 (+ 5140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:17.298905 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:17.298914 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:17.298918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:17.299095 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:17.299575 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:17.299580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:17.299584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:17.300116 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:17.300419 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:17.300833 (+ 414us) server_negotiation.cc:299] Negotiation successful
0923 20:20:17.301081 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:20:17.302414 24028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:17.345110 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:17.344576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:38299 (local address 127.0.0.1:55266)
0923 20:20:17.344800 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:17.344925 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 20:20:17.346536 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:17.346122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:38299 (local address 127.0.0.1:55280)
0923 20:20:17.346258 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:17.346363 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 20:20:17.407558 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:17.406995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:38299 (local address 127.0.0.1:55288)
0923 20:20:17.407204 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:17.407343 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 20:20:17.408874 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:17.408486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:38299 (local address 127.0.0.1:55304)
0923 20:20:17.408654 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:17.408742 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 20:20:17.472684 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 23905
I20260923 20:20:17.489689 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:17.489030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:38299 (local address 127.0.0.1:55320)
0923 20:20:17.489344 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:17.489444 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260923 20:20:17.491146 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:17.490648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:38299 (local address 127.0.0.1:55334)
0923 20:20:17.490920 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:17.491005 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260923 20:20:17.500694 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 24039
Sep 23 20:20:17 dist-test-slave-vl24 krb5kdc[23665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:20:17.527439 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:17.501663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.5.83.67:42239)
0923 20:20:17.505137 (+ 3474us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:17.505144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:17.505165 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:20:17.505246 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:17.505278 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:17.505389 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:17.507247 (+ 1858us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:17.507412 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:17.508665 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:17.511058 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:17.511746 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:17.511871 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:17.518530 (+ 6659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:17.518548 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:17.518577 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:17.518636 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:17.523398 (+ 4762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:17.524343 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:17.524360 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:17.524366 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:17.524751 (+ 385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:17.525242 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:17.525250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:17.525255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:17.525882 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:17.526300 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:17.526587 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 20:20:17.526882 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3332,"thread_start_us":3222,"threads_started":1}
I20260923 20:20:17.528275 24162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:17.651845 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 24039
I20260923 20:20:17.664323 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:17.663679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:38299 (local address 127.0.0.1:55336)
0923 20:20:17.663942 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:17.664071 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 20:20:17.665622 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:17.665133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:38299 (local address 127.0.0.1:55346)
0923 20:20:17.665336 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:17.665438 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:38299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 20:20:17.674770 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 23678
I20260923 20:20:17.675875 23740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:17.787539 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 23678
2026-09-23T20:20:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4610 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T20:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:17Z Disabled control of system clock
I20260923 20:20:17.870162 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36975
--webserver_interface=127.5.83.126
--webserver_port=0
--builtin_ntp_servers=127.5.83.84:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:36975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:20:18.213851 24222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:18.214148 24222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:18.214231 24222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:18.221608 24222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:18.221714 24222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:18.221752 24222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:18.221802 24222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:18.221848 24222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:18.230883 24222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:36975
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:18.232214 24222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:18.234014 24222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:18.246867 24228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:18.246800 24227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:18.246933 24230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:18.248039 24222 server_base.cc:1034] running on GCE node
I20260923 20:20:18.248869 24222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:18.250311 24222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:18.251555 24222 hybrid_clock.cc:648] HybridClock initialized: now 1790194818251531 us; error 54 us; skew 500 ppm
I20260923 20:20:18.251930 24222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:20:18.257863 24222 webserver.cc:460] Webserver started at http://127.5.83.126:38067/ using document root <none> and password file <none>
I20260923 20:20:18.258617 24222 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:18.258755 24222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:18.259088 24222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:18.262109 24222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "d23f2144ab2d490993bc1971dc3baffe"
format_stamp: "Formatted at 2026-09-23 20:20:18 on dist-test-slave-vl24"
server_key: "4e5a9799fc1ea91b8627b5e2a965e9e6"
server_key_iv: "02264768fd50478db5b6f67e766aa0af"
server_key_version: "encryptionkey@0"
I20260923 20:20:18.262912 24222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "d23f2144ab2d490993bc1971dc3baffe"
format_stamp: "Formatted at 2026-09-23 20:20:18 on dist-test-slave-vl24"
server_key: "4e5a9799fc1ea91b8627b5e2a965e9e6"
server_key_iv: "02264768fd50478db5b6f67e766aa0af"
server_key_version: "encryptionkey@0"
I20260923 20:20:18.268146 24222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:20:18.271639 24236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:18.273106 24222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:20:18.273294 24222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "d23f2144ab2d490993bc1971dc3baffe"
format_stamp: "Formatted at 2026-09-23 20:20:18 on dist-test-slave-vl24"
server_key: "4e5a9799fc1ea91b8627b5e2a965e9e6"
server_key_iv: "02264768fd50478db5b6f67e766aa0af"
server_key_version: "encryptionkey@0"
I20260923 20:20:18.273522 24222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:18.284421 24222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:18.285267 24222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:18.285560 24222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:18.304447 24222 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:36975
I20260923 20:20:18.304467 24287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:36975 every 8 connection(s)
I20260923 20:20:18.305857 24222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:18.307004 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 24222
I20260923 20:20:18.307188 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:18.307662 5453 external_mini_cluster.cc:1428] Setting key 6470bdb3d6348331ac0d9fc8834fc3cc
I20260923 20:20:18.310267 24288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:18.321694 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:18.309550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.5.83.126:36975)
0923 20:20:18.310602 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:18.310612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:18.310651 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:20:18.312238 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:18.312259 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:18.312788 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:18.313115 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:18.313740 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:18.315091 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:18.317118 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:18.317813 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:18.317971 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:18.319231 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:18.319238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:18.319257 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:18.319855 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:18.319933 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:18.319969 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:18.320064 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:18.320302 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 20:20:18.320721 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":404,"threads_started":1}
I20260923 20:20:18.323232 24288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe: Bootstrap starting.
I20260923 20:20:18.326699 24288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:18.327796 24288 log.cc:824] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:18.330724 24288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe: No bootstrap required, opened a new log
I20260923 20:20:18.338071 24288 raft_consensus.cc:348] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23f2144ab2d490993bc1971dc3baffe" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36975 } }
I20260923 20:20:18.338449 24288 raft_consensus.cc:374] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:18.338555 24288 raft_consensus.cc:729] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23f2144ab2d490993bc1971dc3baffe, State: Initialized, Role: FOLLOWER
I20260923 20:20:18.339226 24288 consensus_queue.cc:260] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23f2144ab2d490993bc1971dc3baffe" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36975 } }
I20260923 20:20:18.339461 24288 raft_consensus.cc:388] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:18.339602 24288 raft_consensus.cc:482] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:18.339775 24288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:18.342072 24288 raft_consensus.cc:504] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23f2144ab2d490993bc1971dc3baffe" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36975 } }
I20260923 20:20:18.342645 24288 leader_election.cc:302] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d23f2144ab2d490993bc1971dc3baffe; no voters:
I20260923 20:20:18.343222 24288 leader_election.cc:288] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:18.343327 24293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:18.344069 24293 raft_consensus.cc:686] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [term 1 LEADER]: Becoming Leader. State: Replica: d23f2144ab2d490993bc1971dc3baffe, State: Running, Role: LEADER
I20260923 20:20:18.344640 24293 consensus_queue.cc:237] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23f2144ab2d490993bc1971dc3baffe" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36975 } }
I20260923 20:20:18.345436 24288 sys_catalog.cc:564] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:18.347573 24295 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d23f2144ab2d490993bc1971dc3baffe. Latest consensus state: current_term: 1 leader_uuid: "d23f2144ab2d490993bc1971dc3baffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23f2144ab2d490993bc1971dc3baffe" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36975 } } }
I20260923 20:20:18.347982 24295 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:18.347718 24294 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d23f2144ab2d490993bc1971dc3baffe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23f2144ab2d490993bc1971dc3baffe" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36975 } } }
I20260923 20:20:18.348343 24294 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:18.354029 24299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:18.359555 24299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:18.368739 24299 catalog_manager.cc:1347] Generated new cluster ID: 65b80e52225e433ba2f01f51406d8a8d
I20260923 20:20:18.368871 24299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:18.399780 24299 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:18.401662 24299 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:18.417690 24299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe: Generated new TSK 0
I20260923 20:20:18.418778 24299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:18.423228 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36975
--builtin_ntp_servers=127.5.83.84:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:20:18.766688 24312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:18.766987 24312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:18.767135 24312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:18.774904 24312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:18.775019 24312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:18.775185 24312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:20:18.784749 24312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:18.786069 24312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:18.787887 24312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:18.801541 24317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:18.801539 24320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:18.801575 24318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:18.802469 24312 server_base.cc:1034] running on GCE node
I20260923 20:20:18.803150 24312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:18.804118 24312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:18.805367 24312 hybrid_clock.cc:648] HybridClock initialized: now 1790194818805299 us; error 52 us; skew 500 ppm
I20260923 20:20:18.805707 24312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:20:18.808017 24312 webserver.cc:460] Webserver started at http://127.5.83.65:37619/ using document root <none> and password file <none>
I20260923 20:20:18.808791 24312 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:18.808923 24312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:18.809253 24312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:18.812176 24312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "20c721dd7d53426a91a0e553f3822545"
format_stamp: "Formatted at 2026-09-23 20:20:18 on dist-test-slave-vl24"
server_key: "0a8179d856a374a95aa42eb604ae7a3f"
server_key_iv: "1233454f7d5ccae08f3ab0fdcb7e1685"
server_key_version: "encryptionkey@0"
I20260923 20:20:18.812999 24312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "20c721dd7d53426a91a0e553f3822545"
format_stamp: "Formatted at 2026-09-23 20:20:18 on dist-test-slave-vl24"
server_key: "0a8179d856a374a95aa42eb604ae7a3f"
server_key_iv: "1233454f7d5ccae08f3ab0fdcb7e1685"
server_key_version: "encryptionkey@0"
I20260923 20:20:18.818150 24312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:20:18.821928 24326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:18.823340 24312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:20:18.823524 24312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "20c721dd7d53426a91a0e553f3822545"
format_stamp: "Formatted at 2026-09-23 20:20:18 on dist-test-slave-vl24"
server_key: "0a8179d856a374a95aa42eb604ae7a3f"
server_key_iv: "1233454f7d5ccae08f3ab0fdcb7e1685"
server_key_version: "encryptionkey@0"
I20260923 20:20:18.823803 24312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:18.839079 24312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:18.839943 24312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:18.840251 24312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:18.841295 24312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:18.842778 24312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:18.842904 24312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:18.843032 24312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:18.843119 24312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:18.877449 24312 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:45365
I20260923 20:20:18.877483 24438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:45365 every 8 connection(s)
I20260923 20:20:18.878741 24312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:20:18.881227 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 24312
I20260923 20:20:18.881455 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:20:18.882229 5453 external_mini_cluster.cc:1428] Setting key 20ab53f27c895e83708e049c2e845015
I20260923 20:20:18.885321 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36975
--builtin_ntp_servers=127.5.83.84:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:20:18.892498 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:18.881217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:36845 (local address 127.5.83.126:36975)
0923 20:20:18.883307 (+ 2090us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:18.883313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:18.883332 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:20:18.885448 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:18.885455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:18.885558 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:18.885675 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:18.886678 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:18.887833 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:18.889419 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:18.890138 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:18.890292 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:18.891546 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:18.891555 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:18.891563 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:18.891590 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:18.891644 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:18.891663 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:18.891760 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:18.891954 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 20:20:18.892069 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1962,"thread_start_us":1862,"threads_started":1}
I20260923 20:20:18.893292 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:18.881778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36975 (local address 127.5.83.65:36845)
0923 20:20:18.883153 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:18.883205 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:18.885183 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:18.885867 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:18.885876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:18.885969 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:18.886513 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:18.886529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:18.887973 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:18.887977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:18.889279 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:18.889290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:18.890334 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:18.890339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:18.890534 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:18.891237 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:20:18.891240 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:18.891279 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:20:18.891354 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:18.891761 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:18.891769 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:18.891775 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 20:20:18.891910 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 20:20:18.892199 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":362,"threads_started":1}
I20260923 20:20:18.894822 24439 heartbeater.cc:347] Connected to a master server at 127.5.83.126:36975
I20260923 20:20:18.895165 24439 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:18.896176 24439 heartbeater.cc:511] Master 127.5.83.126:36975 requested a full tablet report, sending...
I20260923 20:20:18.898273 24253 ts_manager.cc:194] Registered new tserver with Master: 20c721dd7d53426a91a0e553f3822545 (127.5.83.65:45365)
I20260923 20:20:18.899798 24253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.65:36845
W20260923 20:20:19.234439 24443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:19.234747 24443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:19.234936 24443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:19.242753 24443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:19.242857 24443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:19.243068 24443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:20:19.252848 24443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:19.254284 24443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:19.256286 24443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:19.270426 24449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:19.270393 24451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:19.270373 24448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:19.271479 24443 server_base.cc:1034] running on GCE node
I20260923 20:20:19.272089 24443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:19.273180 24443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:19.274434 24443 hybrid_clock.cc:648] HybridClock initialized: now 1790194819274376 us; error 45 us; skew 500 ppm
I20260923 20:20:19.274785 24443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:20:19.277237 24443 webserver.cc:460] Webserver started at http://127.5.83.66:41841/ using document root <none> and password file <none>
I20260923 20:20:19.278003 24443 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:19.278113 24443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:19.278474 24443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:19.281410 24443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "06a00e4599e244fa80f9f84eb88fc499"
format_stamp: "Formatted at 2026-09-23 20:20:19 on dist-test-slave-vl24"
server_key: "ad52d5c4443ad109f6c658c741ed1aa1"
server_key_iv: "fea6c99db266446d7e1b37fa5533a972"
server_key_version: "encryptionkey@0"
I20260923 20:20:19.282204 24443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "06a00e4599e244fa80f9f84eb88fc499"
format_stamp: "Formatted at 2026-09-23 20:20:19 on dist-test-slave-vl24"
server_key: "ad52d5c4443ad109f6c658c741ed1aa1"
server_key_iv: "fea6c99db266446d7e1b37fa5533a972"
server_key_version: "encryptionkey@0"
I20260923 20:20:19.287382 24443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:20:19.290892 24457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:19.292270 24443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:20:19.292452 24443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "06a00e4599e244fa80f9f84eb88fc499"
format_stamp: "Formatted at 2026-09-23 20:20:19 on dist-test-slave-vl24"
server_key: "ad52d5c4443ad109f6c658c741ed1aa1"
server_key_iv: "fea6c99db266446d7e1b37fa5533a972"
server_key_version: "encryptionkey@0"
I20260923 20:20:19.292732 24443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:19.308243 24443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:19.309154 24443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:19.309424 24443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:19.310464 24443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:19.311900 24443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:19.312016 24443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:19.312130 24443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:19.312216 24443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:19.346414 24443 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:35045
I20260923 20:20:19.346493 24569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:35045 every 8 connection(s)
I20260923 20:20:19.347884 24443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:20:19.354913 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 24443
I20260923 20:20:19.355096 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:20:19.355610 5453 external_mini_cluster.cc:1428] Setting key 8778ffee6e10fb23dcec72ed6bc7308b
I20260923 20:20:19.358383 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36975
--builtin_ntp_servers=127.5.83.84:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:20:19.360713 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:19.350603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:56157 (local address 127.5.83.126:36975)
0923 20:20:19.350788 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:19.350793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:19.352199 (+ 1406us) server_negotiation.cc:407] Connection header received
0923 20:20:19.354047 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:19.354053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:19.354147 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:19.354279 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:19.355487 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.356426 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.358005 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.358398 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.358533 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:19.359885 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:19.359891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:19.359896 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:19.359923 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:19.359953 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:19.359965 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:19.360060 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:19.360283 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 20:20:19.360381 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:20:19.361740 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:19.351094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36975 (local address 127.5.83.66:56157)
0923 20:20:19.352011 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:19.352055 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:19.353769 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:19.354508 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:19.354516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:19.354601 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:19.355315 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:19.355334 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.356602 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.356608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:19.357829 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:19.357841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.358613 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.358620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:19.358855 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:19.359572 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:20:19.359574 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:19.359613 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:20:19.359685 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:19.360058 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:19.360070 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:19.360080 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 20:20:19.360271 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 20:20:19.360699 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":305,"threads_started":1}
I20260923 20:20:19.363152 24570 heartbeater.cc:347] Connected to a master server at 127.5.83.126:36975
I20260923 20:20:19.363476 24570 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:19.364449 24570 heartbeater.cc:511] Master 127.5.83.126:36975 requested a full tablet report, sending...
I20260923 20:20:19.366187 24253 ts_manager.cc:194] Registered new tserver with Master: 06a00e4599e244fa80f9f84eb88fc499 (127.5.83.66:35045)
I20260923 20:20:19.367420 24253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.66:56157
W20260923 20:20:19.697608 24573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:19.697885 24573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:19.697973 24573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:19.705245 24573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:19.705368 24573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:19.705488 24573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:20:19.714619 24573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:36975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:19.716005 24573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:19.717780 24573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:19.730996 24579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:19.730962 24581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:19.731262 24578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:19.731674 24573 server_base.cc:1034] running on GCE node
I20260923 20:20:19.732329 24573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:19.733310 24573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:19.734547 24573 hybrid_clock.cc:648] HybridClock initialized: now 1790194819734510 us; error 59 us; skew 500 ppm
I20260923 20:20:19.734871 24573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:20:19.737221 24573 webserver.cc:460] Webserver started at http://127.5.83.67:35363/ using document root <none> and password file <none>
I20260923 20:20:19.737946 24573 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:19.738092 24573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:19.738421 24573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:19.741467 24573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "0fe6a24863fd4ef692c27943770406ff"
format_stamp: "Formatted at 2026-09-23 20:20:19 on dist-test-slave-vl24"
server_key: "86ede86841535bcdb89939549031a5f6"
server_key_iv: "b170c013f490ba1e606f2add7a1af272"
server_key_version: "encryptionkey@0"
I20260923 20:20:19.742305 24573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "0fe6a24863fd4ef692c27943770406ff"
format_stamp: "Formatted at 2026-09-23 20:20:19 on dist-test-slave-vl24"
server_key: "86ede86841535bcdb89939549031a5f6"
server_key_iv: "b170c013f490ba1e606f2add7a1af272"
server_key_version: "encryptionkey@0"
I20260923 20:20:19.747866 24573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:20:19.751708 24587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:19.753233 24573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:20:19.753417 24573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "0fe6a24863fd4ef692c27943770406ff"
format_stamp: "Formatted at 2026-09-23 20:20:19 on dist-test-slave-vl24"
server_key: "86ede86841535bcdb89939549031a5f6"
server_key_iv: "b170c013f490ba1e606f2add7a1af272"
server_key_version: "encryptionkey@0"
I20260923 20:20:19.753664 24573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:19.779582 24573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:19.780475 24573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:19.780923 24573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:19.781872 24573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:19.783313 24573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:19.783402 24573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:19.783486 24573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:19.783545 24573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:19.817127 24573 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:38825
I20260923 20:20:19.817291 24699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:38825 every 8 connection(s)
I20260923 20:20:19.818454 24573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:20:19.827711 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 24573
I20260923 20:20:19.827888 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:20:19.828377 5453 external_mini_cluster.cc:1428] Setting key acc7c2426b7971e792b3137eba1b8fdc
I20260923 20:20:19.830900 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:19.820937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:55519 (local address 127.5.83.126:36975)
0923 20:20:19.821142 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:19.821147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:19.822468 (+ 1321us) server_negotiation.cc:407] Connection header received
0923 20:20:19.824151 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:19.824156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:19.824227 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:19.824321 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:19.825419 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.826163 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.828117 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.828620 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.828719 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:19.830154 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:19.830165 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:19.830171 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:19.830191 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:19.830229 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:19.830247 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:19.830339 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:19.830575 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 20:20:19.830648 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 20:20:19.832031 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:19.821390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36975 (local address 127.5.83.67:55519)
0923 20:20:19.822308 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:19.822343 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:19.823925 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:19.824640 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:19.824649 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:19.824739 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:19.825270 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:19.825286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.826283 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.826286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:19.827954 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:19.827971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.828852 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.828860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:19.829065 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:19.829798 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:20:19.829800 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:19.829839 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:20:19.829951 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:19.830355 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:19.830367 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:19.830375 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 20:20:19.830563 (+ 188us) client_negotiation.cc:240] Negotiation successful
0923 20:20:19.830954 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":309,"threads_started":1}
I20260923 20:20:19.833343 24700 heartbeater.cc:347] Connected to a master server at 127.5.83.126:36975
I20260923 20:20:19.833663 24700 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:19.834373 24700 heartbeater.cc:511] Master 127.5.83.126:36975 requested a full tablet report, sending...
I20260923 20:20:19.835676 24253 ts_manager.cc:194] Registered new tserver with Master: 0fe6a24863fd4ef692c27943770406ff (127.5.83.67:38825)
I20260923 20:20:19.836576 24253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.67:55519
I20260923 20:20:19.843616 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:20:19.853739 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:19.847988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.5.83.126:36975)
0923 20:20:19.848172 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:19.848176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:19.848502 (+ 326us) server_negotiation.cc:407] Connection header received
0923 20:20:19.848741 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:19.848747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:19.848852 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:19.848971 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:19.849439 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.850119 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.851567 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.852001 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.852090 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:19.853148 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:19.853153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:19.853156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:19.853173 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:19.853203 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:19.853213 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:19.853318 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:19.853470 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 20:20:19.853516 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:20:19.857718 24253 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:20:19.864904 24253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:20:19.892566 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:19.877070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.5.83.65:45365)
0923 20:20:19.877987 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:19.877998 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:19.878715 (+ 717us) server_negotiation.cc:407] Connection header received
0923 20:20:19.879101 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:19.879109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:19.879360 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:19.879520 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:19.880588 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.882637 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.886720 (+ 4083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.888777 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.888922 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:19.889319 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:19.891729 (+ 2410us) server_negotiation.cc:299] Negotiation successful
0923 20:20:19.892011 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":1085,"threads_started":1}
I20260923 20:20:19.893777 24712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:19.876836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:45365 (local address 127.0.0.1:55532)
0923 20:20:19.878588 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:19.878623 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:19.878861 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:19.879860 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:19.879867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:19.879900 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:19.880369 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:19.880384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.882825 (+ 2441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.882830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:19.886438 (+ 3608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:19.886457 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.891027 (+ 4570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.891037 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:19.891355 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:19.891436 (+ 81us) client_negotiation.cc:769] Sending connection context
0923 20:20:19.892988 (+ 1552us) client_negotiation.cc:240] Negotiation successful
0923 20:20:19.893311 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1499,"thread_start_us":1294,"threads_started":1}
I20260923 20:20:19.899369 24374 tablet_service.cc:1467] Processing CreateTablet for tablet b03e214f365c40c9bbacabf219ecabbb (DEFAULT_TABLE table=test-table [id=1ee8d13db6ed409791bb5914f36dd3ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:19.901021 24374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b03e214f365c40c9bbacabf219ecabbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:19.902652 24439 heartbeater.cc:503] Master 127.5.83.126:36975 was elected leader, sending a full tablet report...
I20260923 20:20:19.903126 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:19.877502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:35045 (local address 127.0.0.1:40650)
0923 20:20:19.889509 (+ 12007us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:19.889535 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:19.889692 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:19.896107 (+ 6415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:19.896111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:19.896134 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:19.896593 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:19.896605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.898104 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.898112 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:19.900304 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:19.900317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.902221 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.902228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:19.902429 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:19.902464 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 20:20:19.902549 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 20:20:19.902683 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11818,"thread_start_us":11691,"threads_started":1}
I20260923 20:20:19.903731 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:19.877788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.5.83.66:35045)
0923 20:20:19.895237 (+ 17449us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:19.895253 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:19.895287 (+ 34us) server_negotiation.cc:407] Connection header received
0923 20:20:19.895417 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:19.895429 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:19.895745 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:19.895968 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:19.896880 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.897964 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.900569 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.901993 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.902136 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:19.903119 (+ 983us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:19.903202 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 20:20:19.903370 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17285,"thread_start_us":17183,"threads_started":1}
I20260923 20:20:19.905335 24716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:19.877837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:38825 (local address 127.0.0.1:37980)
0923 20:20:19.895795 (+ 17958us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:19.895821 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:19.895965 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:19.896558 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:19.896567 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:19.896594 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:19.898006 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:19.898019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.899992 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.899998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:19.902113 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:19.902127 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.904761 (+ 2634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.904769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:19.904957 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:19.904981 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 20:20:19.905048 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 20:20:19.905136 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17834,"thread_start_us":17691,"threads_started":1}
I20260923 20:20:19.905894 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:19.880912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37980 (local address 127.5.83.67:38825)
0923 20:20:19.894604 (+ 13692us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:19.894612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:19.895980 (+ 1368us) server_negotiation.cc:407] Connection header received
0923 20:20:19.896086 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:19.896094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:19.896314 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:19.896559 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:19.898202 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.899812 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.902340 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:19.904502 (+ 2162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:19.904684 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:19.904955 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:19.905146 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 20:20:19.905317 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13478,"thread_start_us":13721,"threads_started":1}
I20260923 20:20:19.907799 24505 tablet_service.cc:1467] Processing CreateTablet for tablet b03e214f365c40c9bbacabf219ecabbb (DEFAULT_TABLE table=test-table [id=1ee8d13db6ed409791bb5914f36dd3ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:19.909323 24505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b03e214f365c40c9bbacabf219ecabbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:19.909564 24635 tablet_service.cc:1467] Processing CreateTablet for tablet b03e214f365c40c9bbacabf219ecabbb (DEFAULT_TABLE table=test-table [id=1ee8d13db6ed409791bb5914f36dd3ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:19.910866 24635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b03e214f365c40c9bbacabf219ecabbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:19.915299 24718 tablet_bootstrap.cc:492] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545: Bootstrap starting.
I20260923 20:20:19.918514 24718 tablet_bootstrap.cc:654] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:19.919546 24718 log.cc:824] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:19.922333 24718 tablet_bootstrap.cc:492] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545: No bootstrap required, opened a new log
I20260923 20:20:19.922612 24718 ts_tablet_manager.cc:1397] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260923 20:20:19.924327 24720 tablet_bootstrap.cc:492] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499: Bootstrap starting.
I20260923 20:20:19.928388 24720 tablet_bootstrap.cc:654] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:19.929702 24720 log.cc:824] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:19.931143 24721 tablet_bootstrap.cc:492] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff: Bootstrap starting.
I20260923 20:20:19.933511 24720 tablet_bootstrap.cc:492] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499: No bootstrap required, opened a new log
I20260923 20:20:19.933444 24718 raft_consensus.cc:348] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c721dd7d53426a91a0e553f3822545" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 45365 } } peers { permanent_uuid: "06a00e4599e244fa80f9f84eb88fc499" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35045 } } peers { permanent_uuid: "0fe6a24863fd4ef692c27943770406ff" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38825 } }
I20260923 20:20:19.933786 24720 ts_tablet_manager.cc:1397] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 20:20:19.933897 24718 raft_consensus.cc:374] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:19.934000 24718 raft_consensus.cc:729] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c721dd7d53426a91a0e553f3822545, State: Initialized, Role: FOLLOWER
I20260923 20:20:19.934708 24718 consensus_queue.cc:260] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c721dd7d53426a91a0e553f3822545" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 45365 } } peers { permanent_uuid: "06a00e4599e244fa80f9f84eb88fc499" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35045 } } peers { permanent_uuid: "0fe6a24863fd4ef692c27943770406ff" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38825 } }
I20260923 20:20:19.935022 24721 tablet_bootstrap.cc:654] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:19.936172 24721 log.cc:824] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:19.936455 24718 ts_tablet_manager.cc:1428] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260923 20:20:19.939471 24721 tablet_bootstrap.cc:492] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff: No bootstrap required, opened a new log
I20260923 20:20:19.939734 24721 ts_tablet_manager.cc:1397] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 20:20:19.945591 24720 raft_consensus.cc:348] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c721dd7d53426a91a0e553f3822545" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 45365 } } peers { permanent_uuid: "06a00e4599e244fa80f9f84eb88fc499" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35045 } } peers { permanent_uuid: "0fe6a24863fd4ef692c27943770406ff" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38825 } }
I20260923 20:20:19.946090 24720 raft_consensus.cc:374] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:19.946194 24720 raft_consensus.cc:729] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a00e4599e244fa80f9f84eb88fc499, State: Initialized, Role: FOLLOWER
I20260923 20:20:19.946805 24720 consensus_queue.cc:260] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c721dd7d53426a91a0e553f3822545" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 45365 } } peers { permanent_uuid: "06a00e4599e244fa80f9f84eb88fc499" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35045 } } peers { permanent_uuid: "0fe6a24863fd4ef692c27943770406ff" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38825 } }
I20260923 20:20:19.948094 24570 heartbeater.cc:503] Master 127.5.83.126:36975 was elected leader, sending a full tablet report...
I20260923 20:20:19.948642 24720 ts_tablet_manager.cc:1428] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260923 20:20:19.949919 24721 raft_consensus.cc:348] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c721dd7d53426a91a0e553f3822545" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 45365 } } peers { permanent_uuid: "06a00e4599e244fa80f9f84eb88fc499" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35045 } } peers { permanent_uuid: "0fe6a24863fd4ef692c27943770406ff" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38825 } }
I20260923 20:20:19.950309 24721 raft_consensus.cc:374] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:19.950414 24721 raft_consensus.cc:729] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe6a24863fd4ef692c27943770406ff, State: Initialized, Role: FOLLOWER
I20260923 20:20:19.951073 24721 consensus_queue.cc:260] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c721dd7d53426a91a0e553f3822545" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 45365 } } peers { permanent_uuid: "06a00e4599e244fa80f9f84eb88fc499" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35045 } } peers { permanent_uuid: "0fe6a24863fd4ef692c27943770406ff" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38825 } }
I20260923 20:20:19.952195 24700 heartbeater.cc:503] Master 127.5.83.126:36975 was elected leader, sending a full tablet report...
I20260923 20:20:19.952744 24721 ts_tablet_manager.cc:1428] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
W20260923 20:20:20.070196 24701 tablet.cc:2369] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:20:20.099915 24571 tablet.cc:2369] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:20:20.131124 24440 tablet.cc:2369] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:20:20.267702 24723 raft_consensus.cc:482] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:20:20.268090 24723 raft_consensus.cc:504] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c721dd7d53426a91a0e553f3822545" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 45365 } } peers { permanent_uuid: "06a00e4599e244fa80f9f84eb88fc499" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35045 } } peers { permanent_uuid: "0fe6a24863fd4ef692c27943770406ff" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38825 } }
I20260923 20:20:20.270036 24723 leader_election.cc:288] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06a00e4599e244fa80f9f84eb88fc499 (127.5.83.66:35045), 0fe6a24863fd4ef692c27943770406ff (127.5.83.67:38825)
I20260923 20:20:20.280015 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.270554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:40809 (local address 127.5.83.66:35045)
0923 20:20:20.270812 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:20.270819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:20.271725 (+ 906us) server_negotiation.cc:407] Connection header received
0923 20:20:20.272302 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:20.272306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:20.272388 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:20.272458 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:20.273190 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.274260 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.277511 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.278773 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.278912 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:20.279010 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:20.279596 (+ 586us) server_negotiation.cc:299] Negotiation successful
0923 20:20:20.279741 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260923 20:20:20.279994 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.270676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:35405 (local address 127.5.83.67:38825)
0923 20:20:20.270862 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:20.270870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:20.271777 (+ 907us) server_negotiation.cc:407] Connection header received
0923 20:20:20.271964 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:20.271968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:20.272031 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:20.272148 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:20.273118 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.274259 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.277510 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.278773 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.278912 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:20.279012 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:20.279596 (+ 584us) server_negotiation.cc:299] Negotiation successful
0923 20:20:20.279741 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:20:20.280176 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.270455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:35045 (local address 127.5.83.65:40809)
0923 20:20:20.271588 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:20.271611 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:20.271820 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:20.272549 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:20.272557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:20.272589 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:20.273037 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:20.273051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.274404 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.274408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:20.277241 (+ 2833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:20.277258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.279004 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.279017 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:20.279329 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:20.279416 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 20:20:20.279519 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 20:20:20.279670 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":315,"spinlock_wait_cycles":3456,"thread_start_us":553,"threads_started":1}
I20260923 20:20:20.280645 24728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.270465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:38825 (local address 127.5.83.65:35405)
0923 20:20:20.271588 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:20.271617 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:20.271820 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:20.272295 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:20.272299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:20.272325 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:20.272868 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:20.272886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.274402 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.274407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:20.277241 (+ 2834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:20.277258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.279035 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.279042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:20.279333 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:20.279416 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 20:20:20.279519 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 20:20:20.279669 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":415,"spinlock_wait_cycles":24064,"thread_start_us":544,"threads_started":1}
I20260923 20:20:20.281397 24525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b03e214f365c40c9bbacabf219ecabbb" candidate_uuid: "20c721dd7d53426a91a0e553f3822545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a00e4599e244fa80f9f84eb88fc499" is_pre_election: true
I20260923 20:20:20.281654 24655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b03e214f365c40c9bbacabf219ecabbb" candidate_uuid: "20c721dd7d53426a91a0e553f3822545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe6a24863fd4ef692c27943770406ff" is_pre_election: true
I20260923 20:20:20.281982 24525 raft_consensus.cc:2454] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20c721dd7d53426a91a0e553f3822545 in term 0.
I20260923 20:20:20.282166 24655 raft_consensus.cc:2454] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20c721dd7d53426a91a0e553f3822545 in term 0.
I20260923 20:20:20.282791 24328 leader_election.cc:302] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06a00e4599e244fa80f9f84eb88fc499, 0fe6a24863fd4ef692c27943770406ff, 20c721dd7d53426a91a0e553f3822545; no voters:
I20260923 20:20:20.283257 24723 raft_consensus.cc:2781] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:20:20.283422 24723 raft_consensus.cc:482] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:20:20.283543 24723 raft_consensus.cc:3037] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:20.285787 24723 raft_consensus.cc:504] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c721dd7d53426a91a0e553f3822545" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 45365 } } peers { permanent_uuid: "06a00e4599e244fa80f9f84eb88fc499" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35045 } } peers { permanent_uuid: "0fe6a24863fd4ef692c27943770406ff" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38825 } }
I20260923 20:20:20.286480 24723 leader_election.cc:288] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [CANDIDATE]: Term 1 election: Requested vote from peers 06a00e4599e244fa80f9f84eb88fc499 (127.5.83.66:35045), 0fe6a24863fd4ef692c27943770406ff (127.5.83.67:38825)
I20260923 20:20:20.286986 24525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b03e214f365c40c9bbacabf219ecabbb" candidate_uuid: "20c721dd7d53426a91a0e553f3822545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a00e4599e244fa80f9f84eb88fc499"
I20260923 20:20:20.287052 24655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b03e214f365c40c9bbacabf219ecabbb" candidate_uuid: "20c721dd7d53426a91a0e553f3822545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe6a24863fd4ef692c27943770406ff"
I20260923 20:20:20.287230 24525 raft_consensus.cc:3037] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:20.287245 24655 raft_consensus.cc:3037] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:20.290568 24525 raft_consensus.cc:2454] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20c721dd7d53426a91a0e553f3822545 in term 1.
I20260923 20:20:20.290561 24655 raft_consensus.cc:2454] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20c721dd7d53426a91a0e553f3822545 in term 1.
I20260923 20:20:20.291085 24328 leader_election.cc:302] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06a00e4599e244fa80f9f84eb88fc499, 0fe6a24863fd4ef692c27943770406ff, 20c721dd7d53426a91a0e553f3822545; no voters:
I20260923 20:20:20.291409 24723 raft_consensus.cc:2781] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:20.291951 24723 raft_consensus.cc:686] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 1 LEADER]: Becoming Leader. State: Replica: 20c721dd7d53426a91a0e553f3822545, State: Running, Role: LEADER
I20260923 20:20:20.292449 24723 consensus_queue.cc:237] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c721dd7d53426a91a0e553f3822545" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 45365 } } peers { permanent_uuid: "06a00e4599e244fa80f9f84eb88fc499" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35045 } } peers { permanent_uuid: "0fe6a24863fd4ef692c27943770406ff" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38825 } }
I20260923 20:20:20.297885 24253 catalog_manager.cc:5495] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20c721dd7d53426a91a0e553f3822545 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "20c721dd7d53426a91a0e553f3822545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c721dd7d53426a91a0e553f3822545" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 45365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a00e4599e244fa80f9f84eb88fc499" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fe6a24863fd4ef692c27943770406ff" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38825 } health_report { overall_health: UNKNOWN } } }
I20260923 20:20:20.351394 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.345795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.5.83.65:45365)
0923 20:20:20.345993 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:20.345999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:20.346019 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:20:20.346246 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:20.346252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:20.346345 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:20.346511 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:20:20.346945 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.347640 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.349136 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.349557 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.349692 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:20.350323 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:20:20.350795 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:20:20.350993 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:20.351113 (+ 120us) server_negotiation.cc:299] Negotiation successful
0923 20:20:20.351158 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 20:20:20.360477 24655 raft_consensus.cc:1261] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff [term 1 FOLLOWER]: Refusing update from remote peer 20c721dd7d53426a91a0e553f3822545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:20:20.360487 24525 raft_consensus.cc:1261] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499 [term 1 FOLLOWER]: Refusing update from remote peer 20c721dd7d53426a91a0e553f3822545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:20:20.361497 24729 consensus_queue.cc:1035] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe6a24863fd4ef692c27943770406ff" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:20:20.361881 24723 consensus_queue.cc:1035] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a00e4599e244fa80f9f84eb88fc499" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 35045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:20:20.373451 24732 mvcc.cc:205] Tried to move back new op lower bound from 7332637984186183680 to 7332637983922352128. Current Snapshot: MvccSnapshot[applied={T|T < 7332637984186183680}]
I20260923 20:20:20.387907 24253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:20:20.388191 24253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:20:20.391858 24253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d23f2144ab2d490993bc1971dc3baffe: Sending DeleteTablet for 3 replicas of tablet b03e214f365c40c9bbacabf219ecabbb
I20260923 20:20:20.393114 24505 tablet_service.cc:1514] Processing DeleteTablet for tablet b03e214f365c40c9bbacabf219ecabbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:20 UTC) from {username='slave'} at 127.0.0.1:40650
I20260923 20:20:20.393117 24374 tablet_service.cc:1514] Processing DeleteTablet for tablet b03e214f365c40c9bbacabf219ecabbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:20 UTC) from {username='slave'} at 127.0.0.1:55532
I20260923 20:20:20.393349 24635 tablet_service.cc:1514] Processing DeleteTablet for tablet b03e214f365c40c9bbacabf219ecabbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:20 UTC) from {username='slave'} at 127.0.0.1:37980
I20260923 20:20:20.395540 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 24312
I20260923 20:20:20.408704 24744 tablet_replica.cc:330] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff: stopping tablet replica
I20260923 20:20:20.409243 24744 raft_consensus.cc:2229] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:20:20.411710 24744 raft_consensus.cc:2258] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:20.414222 24744 ts_tablet_manager.cc:1905] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:20.421198 24742 tablet_replica.cc:330] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545: stopping tablet replica
I20260923 20:20:20.421720 24742 raft_consensus.cc:2229] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:20:20.422463 24742 raft_consensus.cc:2258] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:20.424342 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.397745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.5.83.65:45365)
0923 20:20:20.398005 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:20.398013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:20.402402 (+ 4389us) server_negotiation.cc:407] Connection header received
0923 20:20:20.403335 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:20.403343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:20.403457 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:20.403542 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:20.404737 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.405987 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.410011 (+ 4024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.410727 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.410832 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:20.421556 (+ 10724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:20.421574 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:20.421601 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:20.422729 (+ 1128us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:20.422824 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:20.422868 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:20.423022 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:20.423590 (+ 568us) server_negotiation.cc:299] Negotiation successful
0923 20:20:20.423761 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:20:20.424801 24742 ts_tablet_manager.cc:1905] T b03e214f365c40c9bbacabf219ecabbb P 20c721dd7d53426a91a0e553f3822545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:20.426397 24434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:20.426846 24743 tablet_replica.cc:330] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499: stopping tablet replica
I20260923 20:20:20.427502 24743 raft_consensus.cc:2229] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:20:20.428382 24743 raft_consensus.cc:2258] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:20.429047 24744 ts_tablet_manager.cc:1918] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:20.429373 24744 log.cc:1196] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/b03e214f365c40c9bbacabf219ecabbb
I20260923 20:20:20.430102 24744 ts_tablet_manager.cc:1939] T b03e214f365c40c9bbacabf219ecabbb P 0fe6a24863fd4ef692c27943770406ff: Deleting consensus metadata
I20260923 20:20:20.431856 24743 ts_tablet_manager.cc:1905] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:20.432675 24237 catalog_manager.cc:4841] TS 0fe6a24863fd4ef692c27943770406ff (127.5.83.67:38825): tablet b03e214f365c40c9bbacabf219ecabbb (table test-table [id=1ee8d13db6ed409791bb5914f36dd3ec]) successfully deleted
I20260923 20:20:20.436987 24743 ts_tablet_manager.cc:1918] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:20.437152 24743 log.cc:1196] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/b03e214f365c40c9bbacabf219ecabbb
I20260923 20:20:20.437588 24743 ts_tablet_manager.cc:1939] T b03e214f365c40c9bbacabf219ecabbb P 06a00e4599e244fa80f9f84eb88fc499: Deleting consensus metadata
I20260923 20:20:20.439229 24238 catalog_manager.cc:4841] TS 06a00e4599e244fa80f9f84eb88fc499 (127.5.83.66:35045): tablet b03e214f365c40c9bbacabf219ecabbb (table test-table [id=1ee8d13db6ed409791bb5914f36dd3ec]) successfully deleted
I20260923 20:20:20.528098 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 24312
W20260923 20:20:20.549937 24239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.65:45365 (error 108)
I20260923 20:20:20.550745 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 24443
I20260923 20:20:20.552171 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.551241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:45365 (local address 127.0.0.1:55562)
0923 20:20:20.551772 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:20.551943 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":231,"threads_started":1}
W20260923 20:20:20.552434 24239 catalog_manager.cc:4584] TS 20c721dd7d53426a91a0e553f3822545 (127.5.83.65:45365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b03e214f365c40c9bbacabf219ecabbb: Network error: Client connection negotiation failed: client connection to 127.5.83.65:45365: connect: Connection refused (error 111)
I20260923 20:20:20.557976 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.551499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.5.83.66:35045)
0923 20:20:20.551717 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:20.551726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:20.551816 (+ 90us) server_negotiation.cc:407] Connection header received
0923 20:20:20.551975 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:20.551981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:20.552083 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:20.552203 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:20.552695 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.553482 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.554865 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.555285 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.555438 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:20.556462 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:20.556484 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:20.556497 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:20.557151 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:20.557197 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:20.557217 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:20.557350 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:20.557484 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 20:20:20.557591 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260923 20:20:20.558637 24565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:20.606264 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.605697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:45365 (local address 127.0.0.1:55576)
0923 20:20:20.605972 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:20.606081 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 20:20:20.607352 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.606951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:45365 (local address 127.0.0.1:55592)
0923 20:20:20.607117 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:20.607198 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:20:20.649169 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 24443
I20260923 20:20:20.668202 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.667310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:45365 (local address 127.0.0.1:55594)
0923 20:20:20.667585 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:20.667725 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260923 20:20:20.668206 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 24573
I20260923 20:20:20.669553 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.668929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:45365 (local address 127.0.0.1:55602)
0923 20:20:20.669259 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:20.669369 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260923 20:20:20.674906 24717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.669054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.5.83.67:38825)
0923 20:20:20.669218 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:20.669225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:20.669246 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:20:20.669402 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:20.669408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:20.669533 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:20.669660 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:20.670069 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.670772 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.671980 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:20.672369 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:20.672468 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:20.673737 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:20.673745 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:20.673759 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:20.674320 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:20.674366 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:20.674384 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:20.674486 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:20.674586 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 20:20:20.674671 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 20:20:20.675534 24695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:20.750778 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.749727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:45365 (local address 127.0.0.1:55612)
0923 20:20:20.750169 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:20.750330 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260923 20:20:20.752223 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:20.751695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:45365 (local address 127.0.0.1:55614)
0923 20:20:20.751946 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:20.752047 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:45365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260923 20:20:20.767424 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 24573
I20260923 20:20:20.784896 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 24222
I20260923 20:20:20.786155 24283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:20.896503 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 24222
2026-09-23T20:20:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3107 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:21 dist-test-slave-vl24 krb5kdc[24754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:21 dist-test-slave-vl24 krb5kdc[24754](info): set up 2 sockets
Sep 23 20:20:21 dist-test-slave-vl24 krb5kdc[24754](info): commencing operation
krb5kdc: starting...
W20260923 20:20:22.010022 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:22 dist-test-slave-vl24 krb5kdc[24754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:20:22.292712 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:37777
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:37777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:22.635947 24767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:22.636241 24767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:22.636334 24767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:22.643739 24767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:22.643877 24767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:22.643935 24767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:22.643992 24767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:22.644037 24767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:22.654274 24767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:37777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:37777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:22.655588 24767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:22.657018 24767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:22.669128 24775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:22.669198 24772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:22.669135 24773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:22.669837 24767 server_base.cc:1034] running on GCE node
I20260923 20:20:22.670642 24767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:22.671964 24767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:22.673162 24767 hybrid_clock.cc:648] HybridClock initialized: now 1790194822673127 us; error 25 us; skew 500 ppm
I20260923 20:20:22.673517 24767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:22 dist-test-slave-vl24 krb5kdc[24754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194822, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:22.676713 24767 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:20:22.678138 24767 webserver.cc:460] Webserver started at http://127.5.83.126:42959/ using document root <none> and password file <none>
I20260923 20:20:22.678799 24767 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:22.678956 24767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:22.679279 24767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:22.682258 24767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "6f72c9de0d334fef844028a53c89cd21"
format_stamp: "Formatted at 2026-09-23 20:20:22 on dist-test-slave-vl24"
server_key: "3c35241bee0884ebf54e93e1cf4ac767"
server_key_iv: "f91bcb4f94bc9e8a84429a61f698c7d1"
server_key_version: "encryptionkey@0"
I20260923 20:20:22.683000 24767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "6f72c9de0d334fef844028a53c89cd21"
format_stamp: "Formatted at 2026-09-23 20:20:22 on dist-test-slave-vl24"
server_key: "3c35241bee0884ebf54e93e1cf4ac767"
server_key_iv: "f91bcb4f94bc9e8a84429a61f698c7d1"
server_key_version: "encryptionkey@0"
I20260923 20:20:22.688110 24767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:20:22.692004 24782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:22.693308 24767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:20:22.693503 24767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "6f72c9de0d334fef844028a53c89cd21"
format_stamp: "Formatted at 2026-09-23 20:20:22 on dist-test-slave-vl24"
server_key: "3c35241bee0884ebf54e93e1cf4ac767"
server_key_iv: "f91bcb4f94bc9e8a84429a61f698c7d1"
server_key_version: "encryptionkey@0"
I20260923 20:20:22.693707 24767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:22.709604 24767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:22.713682 24767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:22.713975 24767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:22.733065 24767 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:37777
I20260923 20:20:22.733089 24833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:37777 every 8 connection(s)
I20260923 20:20:22.734558 24767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:22.738507 24834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:22.740863 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 24767
I20260923 20:20:22.741040 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:22.741602 5453 external_mini_cluster.cc:1428] Setting key 161f0e31c422aec1df64b9cbe560ed4d
I20260923 20:20:22.752352 24834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21: Bootstrap starting.
Sep 23 20:20:22 dist-test-slave-vl24 krb5kdc[24754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:22.755724 24834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:22.756945 24834 log.cc:824] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:22.760277 24834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21: No bootstrap required, opened a new log
I20260923 20:20:22.764392 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:22.743613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.5.83.126:37777)
0923 20:20:22.744458 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:22.744471 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:22.744514 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:20:22.746240 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:22.746268 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:22.746689 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:22.747087 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:22.748639 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:22.750005 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:22.751343 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:22.752071 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:22.752285 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:22.756613 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:22.756626 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:22.756641 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:22.756686 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:22.760153 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:22.761003 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:22.761010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:22.761016 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:22.761215 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:22.761609 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:22.761615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:22.761619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:22.762154 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:22.762500 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:22.762948 (+ 448us) server_negotiation.cc:299] Negotiation successful
0923 20:20:22.763364 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":370,"threads_started":1}
I20260923 20:20:22.768972 24834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f72c9de0d334fef844028a53c89cd21" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37777 } }
I20260923 20:20:22.769361 24834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:22.769467 24834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f72c9de0d334fef844028a53c89cd21, State: Initialized, Role: FOLLOWER
I20260923 20:20:22.770027 24834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f72c9de0d334fef844028a53c89cd21" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37777 } }
I20260923 20:20:22.770280 24834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:22.770432 24834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:22.770606 24834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:22.773418 24834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f72c9de0d334fef844028a53c89cd21" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37777 } }
I20260923 20:20:22.773989 24834 leader_election.cc:302] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f72c9de0d334fef844028a53c89cd21; no voters:
I20260923 20:20:22.774590 24834 leader_election.cc:288] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:22.774734 24839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:22.775449 24839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [term 1 LEADER]: Becoming Leader. State: Replica: 6f72c9de0d334fef844028a53c89cd21, State: Running, Role: LEADER
I20260923 20:20:22.776098 24839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f72c9de0d334fef844028a53c89cd21" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37777 } }
I20260923 20:20:22.776774 24834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:22.778924 24840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f72c9de0d334fef844028a53c89cd21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f72c9de0d334fef844028a53c89cd21" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37777 } } }
I20260923 20:20:22.779323 24840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:22.780139 24841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f72c9de0d334fef844028a53c89cd21. Latest consensus state: current_term: 1 leader_uuid: "6f72c9de0d334fef844028a53c89cd21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f72c9de0d334fef844028a53c89cd21" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 37777 } } }
I20260923 20:20:22.780611 24841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:22.782645 24845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:22.788291 24845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:22.798637 24845 catalog_manager.cc:1347] Generated new cluster ID: fef7cfbffe464e15a8d5d3afef2b156a
I20260923 20:20:22.798772 24845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:22.816890 24845 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:22.818189 24845 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:22.825738 24845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21: Generated new TSK 0
I20260923 20:20:22.826618 24845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.83.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
Entry for principal kudu/127.5.83.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab.
I20260923 20:20:22.908874 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37777
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:23.262413 24860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:23.262699 24860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:23.262831 24860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:23.270659 24860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:23.270781 24860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:23.270896 24860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:20:23.281085 24860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.65.keytab
--principal=kudu/127.5.83.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:23.282460 24860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:23.283771 24860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:23.297665 24868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:23.297824 24866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:23.297719 24865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:23.299203 24860 server_base.cc:1034] running on GCE node
I20260923 20:20:23.299916 24860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:23.300972 24860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:23.302199 24860 hybrid_clock.cc:648] HybridClock initialized: now 1790194823302150 us; error 40 us; skew 500 ppm
I20260923 20:20:23.302528 24860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:23 dist-test-slave-vl24 krb5kdc[24754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194823, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:23.308939 24860 init.cc:377] Logged in from keytab as kudu/127.5.83.65@KRBTEST.COM (short username kudu)
I20260923 20:20:23.310570 24860 webserver.cc:460] Webserver started at http://127.5.83.65:33343/ using document root <none> and password file <none>
I20260923 20:20:23.311298 24860 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:23.311446 24860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:23.311766 24860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:23.314826 24860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "10349c6525d74012b91a787a4aee8284"
format_stamp: "Formatted at 2026-09-23 20:20:23 on dist-test-slave-vl24"
server_key: "490c1723cae6a998ccf1103441fd3e7c"
server_key_iv: "2d59f43aa28618f657b9fd233e012126"
server_key_version: "encryptionkey@0"
I20260923 20:20:23.315652 24860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "10349c6525d74012b91a787a4aee8284"
format_stamp: "Formatted at 2026-09-23 20:20:23 on dist-test-slave-vl24"
server_key: "490c1723cae6a998ccf1103441fd3e7c"
server_key_iv: "2d59f43aa28618f657b9fd233e012126"
server_key_version: "encryptionkey@0"
I20260923 20:20:23.321216 24860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:20:23.324963 24875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:23.326404 24860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:20:23.326588 24860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "10349c6525d74012b91a787a4aee8284"
format_stamp: "Formatted at 2026-09-23 20:20:23 on dist-test-slave-vl24"
server_key: "490c1723cae6a998ccf1103441fd3e7c"
server_key_iv: "2d59f43aa28618f657b9fd233e012126"
server_key_version: "encryptionkey@0"
I20260923 20:20:23.326807 24860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:23.343374 24860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:23.347675 24860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:23.347981 24860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:23.348981 24860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:23.350374 24860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:23.350486 24860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:23.350642 24860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:23.350713 24860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:23.385697 24860 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:37699
I20260923 20:20:23.385788 24988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:37699 every 8 connection(s)
I20260923 20:20:23.387033 24860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:20:23.391553 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 24860
I20260923 20:20:23.391752 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:20:23.392179 5453 external_mini_cluster.cc:1428] Setting key 63263d09e0cc83b2e6db3a1e6bd71456
Sep 23 20:20:23 dist-test-slave-vl24 krb5kdc[24754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194823, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.65@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:23.408667 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:23.389699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:33447 (local address 127.5.83.126:37777)
0923 20:20:23.390171 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:23.390176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:23.391178 (+ 1002us) server_negotiation.cc:407] Connection header received
0923 20:20:23.393312 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:23.393317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:23.393415 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:23.393542 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:23.395108 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:23.395830 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:23.397684 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:23.398251 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:23.398386 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:23.401976 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:23.401982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:23.401985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:23.402020 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:23.405096 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:23.406190 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:23.406195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:23.406197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:23.406324 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:23.406856 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:23.406859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:23.406861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:23.407116 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:23.407226 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:23.408159 (+ 933us) server_negotiation.cc:299] Negotiation successful
0923 20:20:23.408326 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":210,"threads_started":1}
I20260923 20:20:23.410305 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:23.390300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:37777 (local address 127.5.83.65:33447)
0923 20:20:23.391010 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:23.391062 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:23.393028 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:23.393809 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:23.393822 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:23.394392 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:23.394936 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:23.394954 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:23.395985 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:23.395997 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:23.397507 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:23.397520 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:23.398479 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:23.398488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:23.398755 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:23.399476 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:20:23.399500 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:23.401728 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:23.405243 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:23.405255 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:23.405277 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:23.405939 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:23.406439 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:23.406447 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:23.406453 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:23.406743 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:23.407239 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:23.407249 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:23.407728 (+ 479us) client_negotiation.cc:769] Sending connection context
0923 20:20:23.408076 (+ 348us) client_negotiation.cc:240] Negotiation successful
0923 20:20:23.408699 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":273,"threads_started":1}
I20260923 20:20:23.411998 24989 heartbeater.cc:347] Connected to a master server at 127.5.83.126:37777
I20260923 20:20:23.412302 24989 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:23.413115 24989 heartbeater.cc:511] Master 127.5.83.126:37777 requested a full tablet report, sending...
I20260923 20:20:23.415162 24799 ts_manager.cc:194] Registered new tserver with Master: 10349c6525d74012b91a787a4aee8284 (127.5.83.65:37699)
I20260923 20:20:23.416673 24799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.65@KRBTEST.COM'} at 127.5.83.65:33447
WARNING: no policy specified for kudu/127.5.83.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
Entry for principal kudu/127.5.83.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab.
I20260923 20:20:23.455897 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37777
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:23.791203 24995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:23.791556 24995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:23.791730 24995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:23.799355 24995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:23.799505 24995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:23.799707 24995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:20:23.809777 24995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.66.keytab
--principal=kudu/127.5.83.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:23.811153 24995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:23.812587 24995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:23.826368 25000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:23.826326 25003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:23.826412 25001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:23.827823 24995 server_base.cc:1034] running on GCE node
I20260923 20:20:23.828409 24995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:23.829422 24995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:23.830669 24995 hybrid_clock.cc:648] HybridClock initialized: now 1790194823830628 us; error 50 us; skew 500 ppm
I20260923 20:20:23.831040 24995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:23 dist-test-slave-vl24 krb5kdc[24754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194823, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:23.834479 24995 init.cc:377] Logged in from keytab as kudu/127.5.83.66@KRBTEST.COM (short username kudu)
I20260923 20:20:23.835949 24995 webserver.cc:460] Webserver started at http://127.5.83.66:36231/ using document root <none> and password file <none>
I20260923 20:20:23.836722 24995 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:23.836839 24995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:23.837100 24995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:23.840054 24995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "c5efbb59aba64033ad7ecd124e4dafd9"
format_stamp: "Formatted at 2026-09-23 20:20:23 on dist-test-slave-vl24"
server_key: "4fd28c87ab665f61b2bd98bb8ef70603"
server_key_iv: "73cd59cff439a19f286cbbb111093e91"
server_key_version: "encryptionkey@0"
I20260923 20:20:23.840893 24995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "c5efbb59aba64033ad7ecd124e4dafd9"
format_stamp: "Formatted at 2026-09-23 20:20:23 on dist-test-slave-vl24"
server_key: "4fd28c87ab665f61b2bd98bb8ef70603"
server_key_iv: "73cd59cff439a19f286cbbb111093e91"
server_key_version: "encryptionkey@0"
I20260923 20:20:23.846040 24995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:20:23.849486 25010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:23.850893 24995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 20:20:23.851037 24995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "c5efbb59aba64033ad7ecd124e4dafd9"
format_stamp: "Formatted at 2026-09-23 20:20:23 on dist-test-slave-vl24"
server_key: "4fd28c87ab665f61b2bd98bb8ef70603"
server_key_iv: "73cd59cff439a19f286cbbb111093e91"
server_key_version: "encryptionkey@0"
I20260923 20:20:23.851214 24995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:23.881916 24995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:23.886493 24995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:23.886749 24995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:23.887743 24995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:23.889191 24995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:23.889309 24995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:23.889478 24995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:23.889539 24995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:23.923192 24995 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:34061
I20260923 20:20:23.923307 25122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:34061 every 8 connection(s)
I20260923 20:20:23.924583 24995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:20:23.924876 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 24995
I20260923 20:20:23.925042 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:20:23.925700 5453 external_mini_cluster.cc:1428] Setting key 65f8a6ad814c754b9897b291a4dd2c29
Sep 23 20:20:23 dist-test-slave-vl24 krb5kdc[24754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194823, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.66@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:23.945353 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:23.927354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:36063 (local address 127.5.83.126:37777)
0923 20:20:23.927847 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:23.927853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:23.929001 (+ 1148us) server_negotiation.cc:407] Connection header received
0923 20:20:23.931002 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:23.931007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:23.931107 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:23.931226 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:23.932827 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:23.933571 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:23.935100 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:23.935548 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:23.935683 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:23.939522 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:23.939526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:23.939530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:23.939562 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:23.942218 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:23.942986 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:23.942991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:23.942993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:23.943127 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:23.943578 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:23.943585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:23.943592 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:23.943862 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:23.944020 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:23.944724 (+ 704us) server_negotiation.cc:299] Negotiation successful
0923 20:20:23.944960 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":250,"threads_started":1}
I20260923 20:20:23.946326 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:23.928028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:37777 (local address 127.5.83.66:36063)
0923 20:20:23.928852 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:23.928890 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:23.930728 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:23.931468 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:23.931477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:23.932026 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:23.932653 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:23.932672 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:23.933718 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:23.933724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:23.934944 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:23.934956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:23.935789 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:23.935797 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:23.936027 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:23.936767 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:20:23.936795 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:23.939285 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:23.942359 (+ 3074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:23.942370 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:23.942387 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:23.942849 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:23.943235 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:23.943241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:23.943244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:23.943433 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:23.943989 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:23.943996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:23.944339 (+ 343us) client_negotiation.cc:769] Sending connection context
0923 20:20:23.944634 (+ 295us) client_negotiation.cc:240] Negotiation successful
0923 20:20:23.945173 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":320,"threads_started":1}
I20260923 20:20:23.947809 25123 heartbeater.cc:347] Connected to a master server at 127.5.83.126:37777
I20260923 20:20:23.948163 25123 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:23.949083 25123 heartbeater.cc:511] Master 127.5.83.126:37777 requested a full tablet report, sending...
I20260923 20:20:23.950682 24799 ts_manager.cc:194] Registered new tserver with Master: c5efbb59aba64033ad7ecd124e4dafd9 (127.5.83.66:34061)
I20260923 20:20:23.951537 24799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.66@KRBTEST.COM'} at 127.5.83.66:36063
WARNING: no policy specified for kudu/127.5.83.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
Entry for principal kudu/127.5.83.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab.
I20260923 20:20:23.989024 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37777
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:24.337853 25129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:24.338151 25129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:24.338249 25129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:24.345568 25129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:24.345682 25129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:24.345891 25129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:20:24.355891 25129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.67.keytab
--principal=kudu/127.5.83.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:37777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:24.357403 25129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:24.358713 25129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:24.372340 25137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:24.372373 25134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:24.372396 25135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:24.373899 25129 server_base.cc:1034] running on GCE node
I20260923 20:20:24.374647 25129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:24.375663 25129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:24.376907 25129 hybrid_clock.cc:648] HybridClock initialized: now 1790194824376841 us; error 59 us; skew 500 ppm
I20260923 20:20:24.377230 25129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:24 dist-test-slave-vl24 krb5kdc[24754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194824, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:24.380358 25129 init.cc:377] Logged in from keytab as kudu/127.5.83.67@KRBTEST.COM (short username kudu)
I20260923 20:20:24.381973 25129 webserver.cc:460] Webserver started at http://127.5.83.67:36215/ using document root <none> and password file <none>
I20260923 20:20:24.382685 25129 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:24.382833 25129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:24.383133 25129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:24.386168 25129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "966fa87d1580447bb7edaa5390256f1b"
format_stamp: "Formatted at 2026-09-23 20:20:24 on dist-test-slave-vl24"
server_key: "19ff6d29e0e75685e89fade1a7f6e05c"
server_key_iv: "8c9e3a8e835d44a858291d03d9854f6b"
server_key_version: "encryptionkey@0"
I20260923 20:20:24.386968 25129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "966fa87d1580447bb7edaa5390256f1b"
format_stamp: "Formatted at 2026-09-23 20:20:24 on dist-test-slave-vl24"
server_key: "19ff6d29e0e75685e89fade1a7f6e05c"
server_key_iv: "8c9e3a8e835d44a858291d03d9854f6b"
server_key_version: "encryptionkey@0"
I20260923 20:20:24.392343 25129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 20:20:24.395851 25144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:24.397433 25129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:20:24.397625 25129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "966fa87d1580447bb7edaa5390256f1b"
format_stamp: "Formatted at 2026-09-23 20:20:24 on dist-test-slave-vl24"
server_key: "19ff6d29e0e75685e89fade1a7f6e05c"
server_key_iv: "8c9e3a8e835d44a858291d03d9854f6b"
server_key_version: "encryptionkey@0"
I20260923 20:20:24.397846 25129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:24.412158 25129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:24.416692 25129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:24.416944 25129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:24.417855 25129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:24.419168 25129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:24.419287 25129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:24.419421 25129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:24.419481 25129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:24.419531 24989 heartbeater.cc:503] Master 127.5.83.126:37777 was elected leader, sending a full tablet report...
I20260923 20:20:24.454573 25129 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:40619
I20260923 20:20:24.454670 25256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:40619 every 8 connection(s)
I20260923 20:20:24.455957 25129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:20:24.457592 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 25129
I20260923 20:20:24.457777 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:20:24.458235 5453 external_mini_cluster.cc:1428] Setting key 33d54703cacd7cafc2b587cb8ddcca76
Sep 23 20:20:24 dist-test-slave-vl24 krb5kdc[24754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194824, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.67@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:24.477541 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.458618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:49361 (local address 127.5.83.126:37777)
0923 20:20:24.459109 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:24.459116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:24.460255 (+ 1139us) server_negotiation.cc:407] Connection header received
0923 20:20:24.462481 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:24.462486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:24.462588 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:24.462719 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:24.464236 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.465113 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.466628 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.467084 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.467223 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.471469 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:24.471476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:24.471481 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:24.471537 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:24.474776 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:24.475430 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:24.475435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:24.475438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:24.475545 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:24.475937 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:24.475940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:24.475942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:24.476192 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:24.476316 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:24.477106 (+ 790us) server_negotiation.cc:299] Negotiation successful
0923 20:20:24.477277 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":226,"threads_started":1}
I20260923 20:20:24.478638 25260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.459206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:37777 (local address 127.5.83.67:49361)
0923 20:20:24.460090 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:24.460152 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:24.462223 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:24.462966 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:24.462974 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:24.463480 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:24.464062 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:24.464079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.465269 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.465275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:24.466485 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:24.466497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.467314 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.467322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:24.467560 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.468177 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:20:24.468203 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:24.471198 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:24.474895 (+ 3697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:24.474902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:24.474915 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:24.475336 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:24.475633 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:20:24.475636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:20:24.475638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:20:24.475843 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:20:24.476296 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:24.476302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:24.476677 (+ 375us) client_negotiation.cc:769] Sending connection context
0923 20:20:24.477017 (+ 340us) client_negotiation.cc:240] Negotiation successful
0923 20:20:24.477403 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":385,"threads_started":1}
I20260923 20:20:24.479908 25257 heartbeater.cc:347] Connected to a master server at 127.5.83.126:37777
I20260923 20:20:24.480170 25257 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:24.480916 25257 heartbeater.cc:511] Master 127.5.83.126:37777 requested a full tablet report, sending...
I20260923 20:20:24.482437 24799 ts_manager.cc:194] Registered new tserver with Master: 966fa87d1580447bb7edaa5390256f1b (127.5.83.67:40619)
I20260923 20:20:24.483343 24799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.83.67@KRBTEST.COM'} at 127.5.83.67:49361
I20260923 20:20:24.486307 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:20:24.503377 25259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.490584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.5.83.126:37777)
0923 20:20:24.490797 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:24.490805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:24.491195 (+ 390us) server_negotiation.cc:407] Connection header received
0923 20:20:24.491421 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:24.491426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:24.491502 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:24.491608 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:24.493115 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.493890 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.495166 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.495590 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.495738 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.498146 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:24.498152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:24.498157 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:24.498196 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:24.500609 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:24.501294 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:24.501301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:24.501304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:24.501447 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:24.501852 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:24.501857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:24.501861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:24.502249 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:24.502510 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:24.502842 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 20:20:24.503070 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:20:24.507615 24799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:20:24.514685 24799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:20:24.558879 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.528001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.5.83.65:37699)
0923 20:20:24.550179 (+ 22178us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:24.550189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:24.550230 (+ 41us) server_negotiation.cc:407] Connection header received
0923 20:20:24.550337 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:24.550345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:24.550616 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:24.551663 (+ 1047us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:24.551867 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.553575 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.556160 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.557537 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.557678 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.557958 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:24.558388 (+ 430us) server_negotiation.cc:299] Negotiation successful
0923 20:20:24.558547 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21963,"thread_start_us":21789,"threads_started":1}
I20260923 20:20:24.558959 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.527074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37699 (local address 127.0.0.1:46656)
0923 20:20:24.548583 (+ 21509us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:24.548653 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:24.548926 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:24.550883 (+ 1957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:24.550893 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:24.550966 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:24.551476 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:24.551489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.553737 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.553742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:24.555994 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:24.556006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.558009 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.558015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:24.558167 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.558199 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:20:24.558344 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 20:20:24.558528 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21137,"mutex_wait_us":5,"spinlock_wait_cycles":21248,"thread_start_us":20940,"threads_started":1}
I20260923 20:20:24.560362 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.527748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:34061 (local address 127.0.0.1:50578)
0923 20:20:24.551349 (+ 23601us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:24.551373 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:24.551504 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:24.552088 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:24.552093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:24.552118 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:24.552708 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:24.552720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.555064 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.555069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:24.557408 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:24.557421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.559447 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.559459 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:24.559716 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.559750 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 20:20:24.559837 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 20:20:24.559960 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23466,"mutex_wait_us":103,"thread_start_us":22503,"threads_started":1}
I20260923 20:20:24.560432 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.528311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.5.83.66:34061)
0923 20:20:24.549543 (+ 21232us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:24.549555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:24.551513 (+ 1958us) server_negotiation.cc:407] Connection header received
0923 20:20:24.551652 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:24.551662 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:24.551876 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:24.552063 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:24.553057 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.554930 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.557589 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.558929 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.559050 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.559233 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:24.559923 (+ 690us) server_negotiation.cc:299] Negotiation successful
0923 20:20:24.560111 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21067,"thread_start_us":20962,"threads_started":1}
I20260923 20:20:24.560668 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.528433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.5.83.67:40619)
0923 20:20:24.537736 (+ 9303us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:24.537752 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:24.550823 (+ 13071us) server_negotiation.cc:407] Connection header received
0923 20:20:24.551318 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:24.551333 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:24.551550 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:24.551713 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:24.553064 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.554601 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.557520 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.558929 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.559050 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.559260 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:24.560127 (+ 867us) server_negotiation.cc:299] Negotiation successful
0923 20:20:24.560310 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9027,"thread_start_us":8816,"threads_started":1}
I20260923 20:20:24.561589 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.527522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:40619 (local address 127.0.0.1:57610)
0923 20:20:24.550632 (+ 23110us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:24.550658 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:24.551001 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:24.552047 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:24.552051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:24.552076 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:24.552746 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:24.552757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.554783 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.554786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:24.557321 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:24.557338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.559590 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.559599 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:24.559915 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.559949 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 20:20:24.560870 (+ 921us) client_negotiation.cc:240] Negotiation successful
0923 20:20:24.561208 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22933,"mutex_wait_us":154,"spinlock_wait_cycles":896,"thread_start_us":23508,"threads_started":1}
I20260923 20:20:24.563467 24923 tablet_service.cc:1467] Processing CreateTablet for tablet e50b534631e241289f10547ebf426ef4 (DEFAULT_TABLE table=test-table [id=56673c301c5d46d1a628c110a8ceded4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:24.564024 25058 tablet_service.cc:1467] Processing CreateTablet for tablet e50b534631e241289f10547ebf426ef4 (DEFAULT_TABLE table=test-table [id=56673c301c5d46d1a628c110a8ceded4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:24.565094 24923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50b534631e241289f10547ebf426ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:24.565198 25058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50b534631e241289f10547ebf426ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:24.565814 25192 tablet_service.cc:1467] Processing CreateTablet for tablet e50b534631e241289f10547ebf426ef4 (DEFAULT_TABLE table=test-table [id=56673c301c5d46d1a628c110a8ceded4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:24.567232 25192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50b534631e241289f10547ebf426ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:24.576261 25276 tablet_bootstrap.cc:492] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9: Bootstrap starting.
I20260923 20:20:24.580411 25276 tablet_bootstrap.cc:654] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:24.581501 25276 log.cc:824] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:24.582739 25278 tablet_bootstrap.cc:492] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b: Bootstrap starting.
I20260923 20:20:24.585294 25277 tablet_bootstrap.cc:492] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284: Bootstrap starting.
I20260923 20:20:24.588021 25278 tablet_bootstrap.cc:654] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:24.588743 25277 tablet_bootstrap.cc:654] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:24.589727 25278 log.cc:824] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:24.589792 25277 log.cc:824] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:24.592733 25276 tablet_bootstrap.cc:492] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9: No bootstrap required, opened a new log
I20260923 20:20:24.592828 25278 tablet_bootstrap.cc:492] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b: No bootstrap required, opened a new log
I20260923 20:20:24.593040 25276 ts_tablet_manager.cc:1397] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260923 20:20:24.593096 25278 ts_tablet_manager.cc:1397] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260923 20:20:24.594208 25277 tablet_bootstrap.cc:492] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284: No bootstrap required, opened a new log
I20260923 20:20:24.594461 25277 ts_tablet_manager.cc:1397] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260923 20:20:24.600900 25276 raft_consensus.cc:348] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10349c6525d74012b91a787a4aee8284" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37699 } } peers { permanent_uuid: "966fa87d1580447bb7edaa5390256f1b" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40619 } } peers { permanent_uuid: "c5efbb59aba64033ad7ecd124e4dafd9" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 34061 } }
I20260923 20:20:24.601296 25276 raft_consensus.cc:374] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:24.601430 25276 raft_consensus.cc:729] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5efbb59aba64033ad7ecd124e4dafd9, State: Initialized, Role: FOLLOWER
I20260923 20:20:24.602061 25276 consensus_queue.cc:260] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10349c6525d74012b91a787a4aee8284" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37699 } } peers { permanent_uuid: "966fa87d1580447bb7edaa5390256f1b" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40619 } } peers { permanent_uuid: "c5efbb59aba64033ad7ecd124e4dafd9" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 34061 } }
I20260923 20:20:24.603343 25123 heartbeater.cc:503] Master 127.5.83.126:37777 was elected leader, sending a full tablet report...
I20260923 20:20:24.604075 25276 ts_tablet_manager.cc:1428] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260923 20:20:24.604017 25278 raft_consensus.cc:348] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10349c6525d74012b91a787a4aee8284" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37699 } } peers { permanent_uuid: "966fa87d1580447bb7edaa5390256f1b" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40619 } } peers { permanent_uuid: "c5efbb59aba64033ad7ecd124e4dafd9" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 34061 } }
I20260923 20:20:24.604472 25278 raft_consensus.cc:374] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:24.604605 25278 raft_consensus.cc:729] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966fa87d1580447bb7edaa5390256f1b, State: Initialized, Role: FOLLOWER
I20260923 20:20:24.605149 25277 raft_consensus.cc:348] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10349c6525d74012b91a787a4aee8284" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37699 } } peers { permanent_uuid: "966fa87d1580447bb7edaa5390256f1b" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40619 } } peers { permanent_uuid: "c5efbb59aba64033ad7ecd124e4dafd9" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 34061 } }
I20260923 20:20:24.605367 25278 consensus_queue.cc:260] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10349c6525d74012b91a787a4aee8284" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37699 } } peers { permanent_uuid: "966fa87d1580447bb7edaa5390256f1b" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40619 } } peers { permanent_uuid: "c5efbb59aba64033ad7ecd124e4dafd9" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 34061 } }
I20260923 20:20:24.605592 25277 raft_consensus.cc:374] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:24.605695 25277 raft_consensus.cc:729] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10349c6525d74012b91a787a4aee8284, State: Initialized, Role: FOLLOWER
I20260923 20:20:24.606604 25257 heartbeater.cc:503] Master 127.5.83.126:37777 was elected leader, sending a full tablet report...
I20260923 20:20:24.606447 25277 consensus_queue.cc:260] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10349c6525d74012b91a787a4aee8284" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37699 } } peers { permanent_uuid: "966fa87d1580447bb7edaa5390256f1b" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40619 } } peers { permanent_uuid: "c5efbb59aba64033ad7ecd124e4dafd9" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 34061 } }
I20260923 20:20:24.607406 25278 ts_tablet_manager.cc:1428] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260923 20:20:24.611332 25277 ts_tablet_manager.cc:1428] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284: Time spent starting tablet: real 0.017s user 0.009s sys 0.009s
W20260923 20:20:24.639460 24990 tablet.cc:2369] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:20:24.676955 25124 tablet.cc:2369] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 20:20:24.707683 25258 tablet.cc:2369] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:20:24.804155 25284 raft_consensus.cc:482] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:20:24.804522 25284 raft_consensus.cc:504] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10349c6525d74012b91a787a4aee8284" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37699 } } peers { permanent_uuid: "966fa87d1580447bb7edaa5390256f1b" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40619 } } peers { permanent_uuid: "c5efbb59aba64033ad7ecd124e4dafd9" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 34061 } }
I20260923 20:20:24.806417 25284 leader_election.cc:288] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 966fa87d1580447bb7edaa5390256f1b (127.5.83.67:40619), c5efbb59aba64033ad7ecd124e4dafd9 (127.5.83.66:34061)
I20260923 20:20:24.815402 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.807040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:40937 (local address 127.5.83.67:40619)
0923 20:20:24.807261 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:24.807268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:24.808230 (+ 962us) server_negotiation.cc:407] Connection header received
0923 20:20:24.808350 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:24.808356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:24.808435 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:24.808579 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:24.809235 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.810318 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.813218 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.814516 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.814658 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.814768 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:24.815048 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 20:20:24.815181 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:20:24.815594 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.806844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:40619 (local address 127.5.83.65:40937)
0923 20:20:24.808110 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:24.808124 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:24.808231 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:24.808582 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:24.808587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:24.808617 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:24.809074 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:24.809083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.810450 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.810455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:24.813038 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:24.813054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.814720 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.814725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:24.814851 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.814876 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 20:20:24.815001 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 20:20:24.815152 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"mutex_wait_us":106,"thread_start_us":591,"threads_started":1}
I20260923 20:20:24.815882 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.807033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:37961 (local address 127.5.83.66:34061)
0923 20:20:24.807252 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:24.807260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:24.807868 (+ 608us) server_negotiation.cc:407] Connection header received
0923 20:20:24.808132 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:24.808138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:24.808236 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:24.808341 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:24.809023 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.810145 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.813556 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.814864 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.815002 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.815097 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:24.815466 (+ 369us) server_negotiation.cc:299] Negotiation successful
0923 20:20:24.815605 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:20:24.816006 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.806720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:34061 (local address 127.5.83.65:37961)
0923 20:20:24.807731 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:24.807750 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:24.807962 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:24.808384 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:24.808387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:24.808406 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:24.808858 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:24.808871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.810292 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.810296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:24.813039 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:24.813054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.815101 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.815107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:24.815281 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.815321 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 20:20:24.815413 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 20:20:24.815540 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":170,"thread_start_us":1097,"threads_started":1}
I20260923 20:20:24.816809 25212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b534631e241289f10547ebf426ef4" candidate_uuid: "10349c6525d74012b91a787a4aee8284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "966fa87d1580447bb7edaa5390256f1b" is_pre_election: true
I20260923 20:20:24.816972 25078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b534631e241289f10547ebf426ef4" candidate_uuid: "10349c6525d74012b91a787a4aee8284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5efbb59aba64033ad7ecd124e4dafd9" is_pre_election: true
I20260923 20:20:24.817457 25078 raft_consensus.cc:2454] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10349c6525d74012b91a787a4aee8284 in term 0.
I20260923 20:20:24.817438 25212 raft_consensus.cc:2454] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10349c6525d74012b91a787a4aee8284 in term 0.
I20260923 20:20:24.818226 24879 leader_election.cc:302] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10349c6525d74012b91a787a4aee8284, 966fa87d1580447bb7edaa5390256f1b, c5efbb59aba64033ad7ecd124e4dafd9; no voters:
I20260923 20:20:24.818711 25284 raft_consensus.cc:2781] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:20:24.818908 25284 raft_consensus.cc:482] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:20:24.819029 25284 raft_consensus.cc:3037] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:24.821370 25284 raft_consensus.cc:504] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10349c6525d74012b91a787a4aee8284" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37699 } } peers { permanent_uuid: "966fa87d1580447bb7edaa5390256f1b" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40619 } } peers { permanent_uuid: "c5efbb59aba64033ad7ecd124e4dafd9" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 34061 } }
I20260923 20:20:24.822042 25284 leader_election.cc:288] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [CANDIDATE]: Term 1 election: Requested vote from peers 966fa87d1580447bb7edaa5390256f1b (127.5.83.67:40619), c5efbb59aba64033ad7ecd124e4dafd9 (127.5.83.66:34061)
I20260923 20:20:24.822578 25212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b534631e241289f10547ebf426ef4" candidate_uuid: "10349c6525d74012b91a787a4aee8284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "966fa87d1580447bb7edaa5390256f1b"
I20260923 20:20:24.822597 25078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b534631e241289f10547ebf426ef4" candidate_uuid: "10349c6525d74012b91a787a4aee8284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5efbb59aba64033ad7ecd124e4dafd9"
I20260923 20:20:24.822808 25212 raft_consensus.cc:3037] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:24.822814 25078 raft_consensus.cc:3037] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:24.825099 25212 raft_consensus.cc:2454] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10349c6525d74012b91a787a4aee8284 in term 1.
I20260923 20:20:24.825101 25078 raft_consensus.cc:2454] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10349c6525d74012b91a787a4aee8284 in term 1.
I20260923 20:20:24.825529 24879 leader_election.cc:302] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10349c6525d74012b91a787a4aee8284, 966fa87d1580447bb7edaa5390256f1b; no voters:
I20260923 20:20:24.825809 25284 raft_consensus.cc:2781] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:24.826251 25284 raft_consensus.cc:686] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 1 LEADER]: Becoming Leader. State: Replica: 10349c6525d74012b91a787a4aee8284, State: Running, Role: LEADER
I20260923 20:20:24.826824 25284 consensus_queue.cc:237] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10349c6525d74012b91a787a4aee8284" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37699 } } peers { permanent_uuid: "966fa87d1580447bb7edaa5390256f1b" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40619 } } peers { permanent_uuid: "c5efbb59aba64033ad7ecd124e4dafd9" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 34061 } }
I20260923 20:20:24.831419 24798 catalog_manager.cc:5495] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10349c6525d74012b91a787a4aee8284 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "10349c6525d74012b91a787a4aee8284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10349c6525d74012b91a787a4aee8284" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "966fa87d1580447bb7edaa5390256f1b" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5efbb59aba64033ad7ecd124e4dafd9" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 34061 } health_report { overall_health: UNKNOWN } } }
I20260923 20:20:24.865425 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.859920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46670 (local address 127.5.83.65:37699)
0923 20:20:24.860188 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:24.860192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:24.860215 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:20:24.860484 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:24.860488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:24.860605 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:24.860726 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:20:24.861089 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.861786 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.863223 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.863609 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.863702 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.864201 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:20:24.864768 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:20:24.864910 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:24.865034 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 20:20:24.865102 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260923 20:20:24.874228 25078 raft_consensus.cc:1261] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9 [term 1 FOLLOWER]: Refusing update from remote peer 10349c6525d74012b91a787a4aee8284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:20:24.874228 25212 raft_consensus.cc:1261] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b [term 1 FOLLOWER]: Refusing update from remote peer 10349c6525d74012b91a787a4aee8284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:20:24.875397 25284 consensus_queue.cc:1035] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "966fa87d1580447bb7edaa5390256f1b" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 40619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:20:24.875763 25287 consensus_queue.cc:1035] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5efbb59aba64033ad7ecd124e4dafd9" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 34061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:20:24.912302 24798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:20:24.912644 24798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:20:24.915944 24798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f72c9de0d334fef844028a53c89cd21: Sending DeleteTablet for 3 replicas of tablet e50b534631e241289f10547ebf426ef4
I20260923 20:20:24.917137 24923 tablet_service.cc:1514] Processing DeleteTablet for tablet e50b534631e241289f10547ebf426ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:24 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:46656
I20260923 20:20:24.917411 25058 tablet_service.cc:1514] Processing DeleteTablet for tablet e50b534631e241289f10547ebf426ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:24 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:50578
I20260923 20:20:24.917424 25192 tablet_service.cc:1514] Processing DeleteTablet for tablet e50b534631e241289f10547ebf426ef4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:24 UTC) from {username='kudu', principal='kudu/127.5.83.126@KRBTEST.COM'} at 127.0.0.1:57610
I20260923 20:20:24.918977 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 24860
I20260923 20:20:24.951761 25302 tablet_replica.cc:330] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b: stopping tablet replica
I20260923 20:20:24.952401 25302 raft_consensus.cc:2229] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:20:24.953088 25302 raft_consensus.cc:2258] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:24.953981 25301 tablet_replica.cc:330] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9: stopping tablet replica
I20260923 20:20:24.954500 25301 raft_consensus.cc:2229] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:20:24.956264 25301 raft_consensus.cc:2258] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:24.959242 25301 ts_tablet_manager.cc:1905] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 20:20:24 dist-test-slave-vl24 krb5kdc[24754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.65@KRBTEST.COM
I20260923 20:20:24.969004 25302 ts_tablet_manager.cc:1905] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:24.976223 25302 ts_tablet_manager.cc:1918] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:24.976430 25302 log.cc:1196] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/e50b534631e241289f10547ebf426ef4
I20260923 20:20:24.977202 25300 tablet_replica.cc:330] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284: stopping tablet replica
I20260923 20:20:24.977367 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:24.920517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46676 (local address 127.5.83.65:37699)
0923 20:20:24.920781 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:24.920791 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:24.943228 (+ 22437us) server_negotiation.cc:407] Connection header received
0923 20:20:24.943783 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:24.943790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:24.943871 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:24.943943 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:24.946198 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.947855 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.955000 (+ 7145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:24.955770 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:24.955905 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:24.968958 (+ 13053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:24.968980 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:24.969000 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:24.969056 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:24.973441 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:24.974401 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:24.974412 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:24.974416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:24.974603 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:24.975095 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:24.975101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:24.975105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:24.975566 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:24.975875 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:24.976542 (+ 667us) server_negotiation.cc:299] Negotiation successful
0923 20:20:24.976897 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:20:24.977768 25300 raft_consensus.cc:2229] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:20:24.978595 25300 raft_consensus.cc:2258] T e50b534631e241289f10547ebf426ef4 P 10349c6525d74012b91a787a4aee8284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:24.979040 24984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:24.980315 25302 ts_tablet_manager.cc:1939] T e50b534631e241289f10547ebf426ef4 P 966fa87d1580447bb7edaa5390256f1b: Deleting consensus metadata
I20260923 20:20:24.982021 24786 catalog_manager.cc:4841] TS 966fa87d1580447bb7edaa5390256f1b (127.5.83.67:40619): tablet e50b534631e241289f10547ebf426ef4 (table test-table [id=56673c301c5d46d1a628c110a8ceded4]) successfully deleted
I20260923 20:20:24.982481 25301 ts_tablet_manager.cc:1918] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:24.982758 25301 log.cc:1196] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/e50b534631e241289f10547ebf426ef4
I20260923 20:20:24.983395 25301 ts_tablet_manager.cc:1939] T e50b534631e241289f10547ebf426ef4 P c5efbb59aba64033ad7ecd124e4dafd9: Deleting consensus metadata
I20260923 20:20:24.985325 24783 catalog_manager.cc:4841] TS c5efbb59aba64033ad7ecd124e4dafd9 (127.5.83.66:34061): tablet e50b534631e241289f10547ebf426ef4 (table test-table [id=56673c301c5d46d1a628c110a8ceded4]) successfully deleted
I20260923 20:20:25.081218 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 24860
I20260923 20:20:25.101437 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 24995
W20260923 20:20:25.101406 24784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.65:37699 (error 108)
I20260923 20:20:25.103510 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:25.102672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37699 (local address 127.0.0.1:46690)
0923 20:20:25.103145 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:25.103287 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":231,"threads_started":1}
W20260923 20:20:25.103789 24784 catalog_manager.cc:4584] TS 10349c6525d74012b91a787a4aee8284 (127.5.83.65:37699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e50b534631e241289f10547ebf426ef4: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37699: connect: Connection refused (error 111)
Sep 23 20:20:25 dist-test-slave-vl24 krb5kdc[24754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.66@KRBTEST.COM
I20260923 20:20:25.118358 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:25.102258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.5.83.66:34061)
0923 20:20:25.102527 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:25.102535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:25.102603 (+ 68us) server_negotiation.cc:407] Connection header received
0923 20:20:25.102871 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:25.102880 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:25.102993 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:25.103113 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:25.104721 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:25.105468 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:25.106705 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:25.107126 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:25.107230 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:25.112035 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:25.112053 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:25.112070 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:25.112113 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:25.115123 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:25.115921 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:25.115928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:25.115930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:25.116071 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:25.116495 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:25.116501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:25.116506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:25.117124 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:25.117435 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:25.117752 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 20:20:25.118001 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260923 20:20:25.118975 25118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:25.157606 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:25.157041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37699 (local address 127.0.0.1:46700)
0923 20:20:25.157286 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:25.157410 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 20:20:25.158684 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:25.158291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37699 (local address 127.0.0.1:46712)
0923 20:20:25.158455 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:25.158535 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:20:25.215200 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 24995
I20260923 20:20:25.219383 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:25.218785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37699 (local address 127.0.0.1:46726)
0923 20:20:25.219041 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:25.219157 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 20:20:25.220579 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:25.220067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37699 (local address 127.0.0.1:46738)
0923 20:20:25.220271 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:25.220379 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 20:20:25.233182 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 25129
Sep 23 20:20:25 dist-test-slave-vl24 krb5kdc[24754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.67@KRBTEST.COM
I20260923 20:20:25.248618 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:25.233997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.5.83.67:40619)
0923 20:20:25.234170 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:25.234174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:25.234193 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:20:25.234302 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:25.234307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:25.234372 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:25.234467 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:25.235744 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:25.236481 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:25.237779 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:25.238172 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:25.238272 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:25.242886 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:25.242901 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:25.242917 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:25.242949 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:25.245741 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:25.246425 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:25.246433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:25.246435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:25.246589 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:25.246975 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:25.246982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:25.246985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:25.247353 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:25.247698 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:25.247972 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 20:20:25.248206 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:20:25.249284 25252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:25.301493 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:25.300985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37699 (local address 127.0.0.1:46750)
0923 20:20:25.301252 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:25.301339 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 20:20:25.302743 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:25.302268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37699 (local address 127.0.0.1:46758)
0923 20:20:25.302468 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:25.302561 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 20:20:25.361007 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 25129
I20260923 20:20:25.378248 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 24767
I20260923 20:20:25.379410 24829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:25.493152 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 24767
2026-09-23T20:20:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4598 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T20:20:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:25Z Disabled control of system clock
I20260923 20:20:25.578050 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:39683
--webserver_interface=127.5.83.126
--webserver_port=0
--builtin_ntp_servers=127.5.83.84:45289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:39683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:20:25.921422 25313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:25.921737 25313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:25.921829 25313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:25.929299 25313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:25.929425 25313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:25.929463 25313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:25.929514 25313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:25.929559 25313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:25.939043 25313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:45289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:39683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:39683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:25.940331 25313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:25.942154 25313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:25.954315 25319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:25.954420 25321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:25.954586 25318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:25.955458 25313 server_base.cc:1034] running on GCE node
I20260923 20:20:25.956270 25313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:25.957803 25313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:25.959033 25313 hybrid_clock.cc:648] HybridClock initialized: now 1790194825958993 us; error 62 us; skew 500 ppm
I20260923 20:20:25.959396 25313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:20:25.961807 25313 webserver.cc:460] Webserver started at http://127.5.83.126:40709/ using document root <none> and password file <none>
I20260923 20:20:25.962504 25313 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:25.962652 25313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:25.963074 25313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:25.966149 25313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "9ab15257d2a941e8b7c6107f7d8d9c0d"
format_stamp: "Formatted at 2026-09-23 20:20:25 on dist-test-slave-vl24"
server_key: "2a909151e5900a92991883818e7a8bd0"
server_key_iv: "3c04722b8f6a484615695fa9ec9d701c"
server_key_version: "encryptionkey@0"
I20260923 20:20:25.966966 25313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "9ab15257d2a941e8b7c6107f7d8d9c0d"
format_stamp: "Formatted at 2026-09-23 20:20:25 on dist-test-slave-vl24"
server_key: "2a909151e5900a92991883818e7a8bd0"
server_key_iv: "3c04722b8f6a484615695fa9ec9d701c"
server_key_version: "encryptionkey@0"
I20260923 20:20:25.972301 25313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 20:20:25.976312 25327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:25.977849 25313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:20:25.978046 25313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "9ab15257d2a941e8b7c6107f7d8d9c0d"
format_stamp: "Formatted at 2026-09-23 20:20:25 on dist-test-slave-vl24"
server_key: "2a909151e5900a92991883818e7a8bd0"
server_key_iv: "3c04722b8f6a484615695fa9ec9d701c"
server_key_version: "encryptionkey@0"
I20260923 20:20:25.978277 25313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:25.994230 25313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:25.995129 25313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:25.995433 25313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:26.015228 25313 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:39683
I20260923 20:20:26.015275 25378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:39683 every 8 connection(s)
I20260923 20:20:26.016661 25313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:26.020661 25379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:26.024384 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 25313
I20260923 20:20:26.024590 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:26.025162 5453 external_mini_cluster.cc:1428] Setting key 00babb7bcfba20b8b332a9aba450a1fa
I20260923 20:20:26.034312 25379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d: Bootstrap starting.
I20260923 20:20:26.038080 25379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:26.039543 25379 log.cc:824] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:26.040128 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:26.027193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.5.83.126:39683)
0923 20:20:26.028196 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:26.028207 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:26.028269 (+ 62us) server_negotiation.cc:407] Connection header received
0923 20:20:26.030073 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:26.030099 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:26.030765 (+ 666us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:26.031243 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:26.031914 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:26.033323 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:26.034802 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:26.035519 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:26.035735 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:26.037041 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:26.037052 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:26.037076 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:26.037813 (+ 737us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:26.037945 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:26.037979 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:26.038139 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:26.038354 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 20:20:26.038745 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":383,"threads_started":1}
I20260923 20:20:26.042791 25379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d: No bootstrap required, opened a new log
I20260923 20:20:26.050474 25379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab15257d2a941e8b7c6107f7d8d9c0d" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39683 } }
I20260923 20:20:26.050920 25379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:26.051038 25379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ab15257d2a941e8b7c6107f7d8d9c0d, State: Initialized, Role: FOLLOWER
I20260923 20:20:26.051744 25379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab15257d2a941e8b7c6107f7d8d9c0d" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39683 } }
I20260923 20:20:26.051992 25379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:26.052145 25379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:26.052330 25379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:26.054661 25379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab15257d2a941e8b7c6107f7d8d9c0d" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39683 } }
I20260923 20:20:26.055199 25379 leader_election.cc:302] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ab15257d2a941e8b7c6107f7d8d9c0d; no voters:
I20260923 20:20:26.055794 25379 leader_election.cc:288] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:26.056020 25384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:26.057469 25384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [term 1 LEADER]: Becoming Leader. State: Replica: 9ab15257d2a941e8b7c6107f7d8d9c0d, State: Running, Role: LEADER
I20260923 20:20:26.058107 25379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:26.057979 25384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab15257d2a941e8b7c6107f7d8d9c0d" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39683 } }
I20260923 20:20:26.061188 25386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ab15257d2a941e8b7c6107f7d8d9c0d. Latest consensus state: current_term: 1 leader_uuid: "9ab15257d2a941e8b7c6107f7d8d9c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab15257d2a941e8b7c6107f7d8d9c0d" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39683 } } }
I20260923 20:20:26.061285 25385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ab15257d2a941e8b7c6107f7d8d9c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab15257d2a941e8b7c6107f7d8d9c0d" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 39683 } } }
I20260923 20:20:26.061590 25386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:26.061704 25385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:26.063413 25392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:26.068740 25392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:26.077318 25392 catalog_manager.cc:1347] Generated new cluster ID: b525d5a05ae54efcb5d18e54fa1e3fe2
I20260923 20:20:26.077453 25392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:26.087472 25392 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:26.088899 25392 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:26.099226 25392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d: Generated new TSK 0
I20260923 20:20:26.100162 25392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:26.111204 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.65:0
--local_ip_for_outbound_sockets=127.5.83.65
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:39683
--builtin_ntp_servers=127.5.83.84:45289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 20:20:26.461700 25403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:26.461977 25403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:26.462127 25403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:26.470168 25403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:26.470269 25403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:26.470456 25403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.65
I20260923 20:20:26.481436 25403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:45289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.83.65
--webserver_port=0
--tserver_master_addrs=127.5.83.126:39683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.65
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:26.482867 25403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:26.484706 25403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:26.498293 25409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:26.498255 25411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:26.498354 25408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:26.502115 25403 server_base.cc:1034] running on GCE node
I20260923 20:20:26.502751 25403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:26.503727 25403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:26.504930 25403 hybrid_clock.cc:648] HybridClock initialized: now 1790194826504887 us; error 34 us; skew 500 ppm
I20260923 20:20:26.505220 25403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:20:26.507493 25403 webserver.cc:460] Webserver started at http://127.5.83.65:37119/ using document root <none> and password file <none>
I20260923 20:20:26.508131 25403 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:26.508214 25403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:26.508664 25403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:26.511549 25403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/data/instance:
uuid: "7c7188a3e9e743058e6201eb8c6ec770"
format_stamp: "Formatted at 2026-09-23 20:20:26 on dist-test-slave-vl24"
server_key: "cdd0685918d26a9116ff2938e03fd841"
server_key_iv: "5f4e46bda80f3ac22d5b47c24d23ded9"
server_key_version: "encryptionkey@0"
I20260923 20:20:26.512280 25403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance:
uuid: "7c7188a3e9e743058e6201eb8c6ec770"
format_stamp: "Formatted at 2026-09-23 20:20:26 on dist-test-slave-vl24"
server_key: "cdd0685918d26a9116ff2938e03fd841"
server_key_iv: "5f4e46bda80f3ac22d5b47c24d23ded9"
server_key_version: "encryptionkey@0"
I20260923 20:20:26.517562 25403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260923 20:20:26.521376 25417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:26.522827 25403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:20:26.523018 25403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/wal
uuid: "7c7188a3e9e743058e6201eb8c6ec770"
format_stamp: "Formatted at 2026-09-23 20:20:26 on dist-test-slave-vl24"
server_key: "cdd0685918d26a9116ff2938e03fd841"
server_key_iv: "5f4e46bda80f3ac22d5b47c24d23ded9"
server_key_version: "encryptionkey@0"
I20260923 20:20:26.523262 25403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:26.540655 25403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:26.541687 25403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:26.541990 25403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:26.543004 25403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:26.544692 25403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:26.544823 25403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:26.544953 25403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:26.545054 25403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:26.580188 25403 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.65:37189
I20260923 20:20:26.580237 25530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.65:37189 every 8 connection(s)
I20260923 20:20:26.581631 25403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/data/info.pb
I20260923 20:20:26.590181 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 25403
I20260923 20:20:26.590395 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-0/wal/instance
I20260923 20:20:26.590863 5453 external_mini_cluster.cc:1428] Setting key e7fa427332f840bb3cd50312ca15f26b
I20260923 20:20:26.594033 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.66:0
--local_ip_for_outbound_sockets=127.5.83.66
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:39683
--builtin_ntp_servers=127.5.83.84:45289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:20:26.595098 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:26.584367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:43077 (local address 127.5.83.126:39683)
0923 20:20:26.584835 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:26.584839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:26.586010 (+ 1171us) server_negotiation.cc:407] Connection header received
0923 20:20:26.587816 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:26.587823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:26.587909 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:26.588031 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:26.589115 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:26.589856 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:26.591666 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:26.592219 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:26.592327 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:26.593979 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:26.593986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:26.593991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:26.594015 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:26.594071 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:26.594090 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:26.594201 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:26.594485 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 20:20:26.594605 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":265,"threads_started":1}
I20260923 20:20:26.595831 25534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:26.584981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:39683 (local address 127.5.83.65:43077)
0923 20:20:26.585851 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:26.585890 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:26.587536 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:26.588247 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:26.588255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:26.588333 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:26.588951 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:26.588967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:26.590008 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:26.590019 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:26.591518 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:26.591532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:26.592464 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:26.592472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:26.592859 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:26.593643 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:20:26.593646 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:26.593688 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:20:26.593773 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:26.594212 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:26.594226 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:26.594237 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:20:26.594409 (+ 172us) client_negotiation.cc:240] Negotiation successful
0923 20:20:26.594736 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":284,"threads_started":1}
I20260923 20:20:26.597442 25531 heartbeater.cc:347] Connected to a master server at 127.5.83.126:39683
I20260923 20:20:26.597762 25531 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:26.598639 25531 heartbeater.cc:511] Master 127.5.83.126:39683 requested a full tablet report, sending...
I20260923 20:20:26.600888 25344 ts_manager.cc:194] Registered new tserver with Master: 7c7188a3e9e743058e6201eb8c6ec770 (127.5.83.65:37189)
I20260923 20:20:26.602398 25344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.65:43077
W20260923 20:20:26.930512 25535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:26.930784 25535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:26.930924 25535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:26.938251 25535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:26.938355 25535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:26.938526 25535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.66
I20260923 20:20:26.948343 25535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:45289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.83.66
--webserver_port=0
--tserver_master_addrs=127.5.83.126:39683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.66
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:26.949714 25535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:26.951606 25535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:26.965341 25540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:26.965400 25541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:26.965638 25543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:26.966589 25535 server_base.cc:1034] running on GCE node
I20260923 20:20:26.967211 25535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:26.968242 25535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:26.969475 25535 hybrid_clock.cc:648] HybridClock initialized: now 1790194826969423 us; error 43 us; skew 500 ppm
I20260923 20:20:26.969794 25535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:20:26.972035 25535 webserver.cc:460] Webserver started at http://127.5.83.66:38361/ using document root <none> and password file <none>
I20260923 20:20:26.972836 25535 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:26.972970 25535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:26.973299 25535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:26.976238 25535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/data/instance:
uuid: "e1fb75c696af4d6e89fe2cd117d39573"
format_stamp: "Formatted at 2026-09-23 20:20:26 on dist-test-slave-vl24"
server_key: "af6cbd9fbe39d0f6f6696b8c00b3edf2"
server_key_iv: "0c741e4b95f4cc7b64cfd7c4ded5ef9c"
server_key_version: "encryptionkey@0"
I20260923 20:20:26.977154 25535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance:
uuid: "e1fb75c696af4d6e89fe2cd117d39573"
format_stamp: "Formatted at 2026-09-23 20:20:26 on dist-test-slave-vl24"
server_key: "af6cbd9fbe39d0f6f6696b8c00b3edf2"
server_key_iv: "0c741e4b95f4cc7b64cfd7c4ded5ef9c"
server_key_version: "encryptionkey@0"
I20260923 20:20:26.982434 25535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:20:26.986095 25550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:26.987409 25535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:20:26.987581 25535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/wal
uuid: "e1fb75c696af4d6e89fe2cd117d39573"
format_stamp: "Formatted at 2026-09-23 20:20:26 on dist-test-slave-vl24"
server_key: "af6cbd9fbe39d0f6f6696b8c00b3edf2"
server_key_iv: "0c741e4b95f4cc7b64cfd7c4ded5ef9c"
server_key_version: "encryptionkey@0"
I20260923 20:20:26.987807 25535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:27.002434 25535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:27.003299 25535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:27.003590 25535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:27.004604 25535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:27.006152 25535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:27.006278 25535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:27.006415 25535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:27.006508 25535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:27.041693 25535 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.66:37193
I20260923 20:20:27.041739 25663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.66:37193 every 8 connection(s)
I20260923 20:20:27.043020 25535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/data/info.pb
I20260923 20:20:27.043423 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 25535
I20260923 20:20:27.043604 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/wal/instance
I20260923 20:20:27.044235 5453 external_mini_cluster.cc:1428] Setting key 854697b59413fadcdc4341a62a99c7d8
I20260923 20:20:27.047659 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.83.67:0
--local_ip_for_outbound_sockets=127.5.83.67
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:39683
--builtin_ntp_servers=127.5.83.84:45289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 20:20:27.058782 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.045845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.66:60225 (local address 127.5.83.126:39683)
0923 20:20:27.046022 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:27.046026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:27.047848 (+ 1822us) server_negotiation.cc:407] Connection header received
0923 20:20:27.050841 (+ 2993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:27.050845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:27.050924 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:27.051002 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:27.052427 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.053182 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.055770 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.056157 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.056256 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.058072 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:27.058077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:27.058080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:27.058100 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:27.058131 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:27.058142 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:27.058211 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:27.058454 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 20:20:27.058536 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:20:27.060030 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.046548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:39683 (local address 127.5.83.66:60225)
0923 20:20:27.047673 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:27.047724 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:27.050505 (+ 2781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:27.051294 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:27.051308 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:27.051430 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:27.052255 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.052283 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.053341 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.053349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.055549 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.055567 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.056398 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.056408 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.056791 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.057560 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:20:27.057565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:27.057621 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:20:27.057753 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:27.058221 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:27.058234 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:27.058245 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 20:20:27.058440 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 20:20:27.058918 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":347,"threads_started":1}
I20260923 20:20:27.061661 25664 heartbeater.cc:347] Connected to a master server at 127.5.83.126:39683
I20260923 20:20:27.062014 25664 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:27.062763 25664 heartbeater.cc:511] Master 127.5.83.126:39683 requested a full tablet report, sending...
I20260923 20:20:27.064359 25344 ts_manager.cc:194] Registered new tserver with Master: e1fb75c696af4d6e89fe2cd117d39573 (127.5.83.66:37193)
I20260923 20:20:27.065717 25344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.66:60225
W20260923 20:20:27.390841 25667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:27.391125 25667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:27.391253 25667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:27.399019 25667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:27.399132 25667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:27.399241 25667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.83.67
I20260923 20:20:27.408486 25667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:45289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.83.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.83.67
--webserver_port=0
--tserver_master_addrs=127.5.83.126:39683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.83.67
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:27.409878 25667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:27.411572 25667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:27.424517 25675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:27.424641 25673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:27.424844 25672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:27.425731 25667 server_base.cc:1034] running on GCE node
I20260923 20:20:27.426398 25667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:27.427409 25667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:27.428642 25667 hybrid_clock.cc:648] HybridClock initialized: now 1790194827428604 us; error 53 us; skew 500 ppm
I20260923 20:20:27.428946 25667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 20:20:27.431166 25667 webserver.cc:460] Webserver started at http://127.5.83.67:37053/ using document root <none> and password file <none>
I20260923 20:20:27.431857 25667 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:27.431993 25667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:27.432319 25667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:27.435217 25667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/data/instance:
uuid: "62979bde45314932a8f12c38f229e39d"
format_stamp: "Formatted at 2026-09-23 20:20:27 on dist-test-slave-vl24"
server_key: "7a271599e8173b839326e75d1d4bdd0e"
server_key_iv: "ca30247b768153388b6cb95a0b629a23"
server_key_version: "encryptionkey@0"
I20260923 20:20:27.436015 25667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance:
uuid: "62979bde45314932a8f12c38f229e39d"
format_stamp: "Formatted at 2026-09-23 20:20:27 on dist-test-slave-vl24"
server_key: "7a271599e8173b839326e75d1d4bdd0e"
server_key_iv: "ca30247b768153388b6cb95a0b629a23"
server_key_version: "encryptionkey@0"
I20260923 20:20:27.441165 25667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260923 20:20:27.444650 25681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:27.445952 25667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:20:27.446126 25667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/wal
uuid: "62979bde45314932a8f12c38f229e39d"
format_stamp: "Formatted at 2026-09-23 20:20:27 on dist-test-slave-vl24"
server_key: "7a271599e8173b839326e75d1d4bdd0e"
server_key_iv: "ca30247b768153388b6cb95a0b629a23"
server_key_version: "encryptionkey@0"
I20260923 20:20:27.446359 25667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:27.471197 25667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:27.472402 25667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:27.472766 25667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:27.474102 25667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 20:20:27.476107 25667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 20:20:27.476229 25667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:27.476346 25667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 20:20:27.476433 25667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:27.514127 25667 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.67:36077
I20260923 20:20:27.514216 25794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.67:36077 every 8 connection(s)
I20260923 20:20:27.515390 25667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/data/info.pb
I20260923 20:20:27.519882 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 25667
I20260923 20:20:27.520066 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/wal/instance
I20260923 20:20:27.520689 5453 external_mini_cluster.cc:1428] Setting key 500d3fb3c23d11a9b90ccd773761f724
I20260923 20:20:27.529042 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.518033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.67:51113 (local address 127.5.83.126:39683)
0923 20:20:27.518259 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:27.518266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:27.519635 (+ 1369us) server_negotiation.cc:407] Connection header received
0923 20:20:27.521983 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:27.521988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:27.522077 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:27.522180 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:27.523445 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.524255 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.526261 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.526832 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.526939 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.528310 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:27.528315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:27.528319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:27.528338 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:27.528371 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:27.528383 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:27.528463 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:27.528721 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 20:20:27.528810 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 20:20:27.530050 25797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.518524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:39683 (local address 127.5.83.67:51113)
0923 20:20:27.519453 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:27.519505 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:27.521702 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:27.522404 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:27.522412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:27.522500 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:20:27.523273 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.523299 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.524436 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.524440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.526112 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.526125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.527039 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.527044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.527247 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.528004 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 20:20:27.528006 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:20:27.528047 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 20:20:27.528128 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:20:27.528475 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:20:27.528482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:20:27.528489 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 20:20:27.528693 (+ 204us) client_negotiation.cc:240] Negotiation successful
0923 20:20:27.528995 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":272,"threads_started":1}
I20260923 20:20:27.531450 25795 heartbeater.cc:347] Connected to a master server at 127.5.83.126:39683
I20260923 20:20:27.531724 25795 heartbeater.cc:464] Registering TS with master...
I20260923 20:20:27.532413 25795 heartbeater.cc:511] Master 127.5.83.126:39683 requested a full tablet report, sending...
I20260923 20:20:27.533846 25344 ts_manager.cc:194] Registered new tserver with Master: 62979bde45314932a8f12c38f229e39d (127.5.83.67:36077)
I20260923 20:20:27.534859 25344 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.83.67:51113
I20260923 20:20:27.536005 5453 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 20:20:27.545874 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.540242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.5.83.126:39683)
0923 20:20:27.540469 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:27.540478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:27.540853 (+ 375us) server_negotiation.cc:407] Connection header received
0923 20:20:27.540979 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:27.540982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:27.541052 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:27.541132 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:27.541607 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.542303 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.543619 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.544034 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.544129 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.545146 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:27.545150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:27.545154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:27.545172 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:27.545203 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:27.545215 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:27.545293 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:27.545424 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 20:20:27.545500 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 20:20:27.549515 25344 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 20:20:27.556412 25344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 20:20:27.579142 25808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.568890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:36077 (local address 127.0.0.1:38604)
0923 20:20:27.570170 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:27.570228 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:27.570435 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:27.571766 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:27.571773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:27.571802 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:27.572218 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.572230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.574307 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.574312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.576219 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.576231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.578217 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.578224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.578411 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.578464 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 20:20:27.578605 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 20:20:27.578792 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":2213,"threads_started":1}
I20260923 20:20:27.579197 25809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.569106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.5.83.67:36077)
0923 20:20:27.571119 (+ 2013us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:27.571131 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:27.571168 (+ 37us) server_negotiation.cc:407] Connection header received
0923 20:20:27.571275 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:27.571284 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:27.571551 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:27.571771 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:27.572451 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.574128 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.576414 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.577959 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.578083 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.578347 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:27.578652 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 20:20:27.578832 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1827,"thread_start_us":1689,"threads_started":1}
I20260923 20:20:27.581516 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.569038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37189 (local address 127.0.0.1:37034)
0923 20:20:27.571382 (+ 2344us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:27.571399 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:27.571505 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:27.573197 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:27.573202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:27.573226 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:27.573565 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.573576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.575336 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.575341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.577529 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.577543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.580665 (+ 3122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.580672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.580857 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.580889 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 20:20:27.580985 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 20:20:27.581117 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2225,"mutex_wait_us":27,"thread_start_us":695,"threads_started":1}
I20260923 20:20:27.582119 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.571744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.5.83.66:37193)
0923 20:20:27.573632 (+ 1888us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:27.573640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:27.573667 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:20:27.573764 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:27.573772 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:27.574274 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:27.574542 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:27.575106 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.576661 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.579096 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.580518 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.580670 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.580878 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:27.581612 (+ 734us) server_negotiation.cc:299] Negotiation successful
0923 20:20:27.581778 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1554,"thread_start_us":1453,"threads_started":1}
I20260923 20:20:27.582652 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.569568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.5.83.65:37189)
0923 20:20:27.572415 (+ 2847us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:27.572423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:27.572449 (+ 26us) server_negotiation.cc:407] Connection header received
0923 20:20:27.572565 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:27.572576 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:27.572801 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:27.572981 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:27.573772 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.575205 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.577772 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.579174 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.579576 (+ 402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.579808 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:27.582078 (+ 2270us) server_negotiation.cc:299] Negotiation successful
0923 20:20:27.582277 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2676,"thread_start_us":2555,"threads_started":1}
I20260923 20:20:27.582710 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.572056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:37193 (local address 127.0.0.1:34072)
0923 20:20:27.572800 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:27.572823 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:27.572974 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:27.574548 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:27.574559 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:27.574581 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:27.574956 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.574964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.576797 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.576802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.578883 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.578896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.581004 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.581009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.581166 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.581196 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 20:20:27.581285 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 20:20:27.581453 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":1,"thread_start_us":443,"threads_started":1}
I20260923 20:20:27.583606 25729 tablet_service.cc:1467] Processing CreateTablet for tablet 720d2280c5134f3c869f60efc927ab25 (DEFAULT_TABLE table=test-table [id=47821c4aa12b48bd9508487ef97b937d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:27.585253 25729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 720d2280c5134f3c869f60efc927ab25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:27.586436 25465 tablet_service.cc:1467] Processing CreateTablet for tablet 720d2280c5134f3c869f60efc927ab25 (DEFAULT_TABLE table=test-table [id=47821c4aa12b48bd9508487ef97b937d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:27.587592 25465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 720d2280c5134f3c869f60efc927ab25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:27.589152 25598 tablet_service.cc:1467] Processing CreateTablet for tablet 720d2280c5134f3c869f60efc927ab25 (DEFAULT_TABLE table=test-table [id=47821c4aa12b48bd9508487ef97b937d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 20:20:27.590461 25598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 720d2280c5134f3c869f60efc927ab25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:27.599078 25813 tablet_bootstrap.cc:492] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770: Bootstrap starting.
I20260923 20:20:27.601286 25814 tablet_bootstrap.cc:492] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d: Bootstrap starting.
I20260923 20:20:27.603008 25813 tablet_bootstrap.cc:654] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:27.604379 25813 log.cc:824] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:27.604961 25814 tablet_bootstrap.cc:654] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:27.606102 25531 heartbeater.cc:503] Master 127.5.83.126:39683 was elected leader, sending a full tablet report...
I20260923 20:20:27.606357 25814 log.cc:824] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:27.606925 25815 tablet_bootstrap.cc:492] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573: Bootstrap starting.
I20260923 20:20:27.608011 25813 tablet_bootstrap.cc:492] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770: No bootstrap required, opened a new log
I20260923 20:20:27.608359 25813 ts_tablet_manager.cc:1397] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260923 20:20:27.610733 25815 tablet_bootstrap.cc:654] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:27.611785 25815 log.cc:824] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:27.616636 25814 tablet_bootstrap.cc:492] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d: No bootstrap required, opened a new log
I20260923 20:20:27.617007 25814 ts_tablet_manager.cc:1397] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20260923 20:20:27.618801 25813 raft_consensus.cc:348] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62979bde45314932a8f12c38f229e39d" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36077 } } peers { permanent_uuid: "7c7188a3e9e743058e6201eb8c6ec770" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37189 } } peers { permanent_uuid: "e1fb75c696af4d6e89fe2cd117d39573" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37193 } }
I20260923 20:20:27.619230 25815 tablet_bootstrap.cc:492] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573: No bootstrap required, opened a new log
I20260923 20:20:27.619249 25813 raft_consensus.cc:374] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:27.619376 25813 raft_consensus.cc:729] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7188a3e9e743058e6201eb8c6ec770, State: Initialized, Role: FOLLOWER
I20260923 20:20:27.619537 25815 ts_tablet_manager.cc:1397] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260923 20:20:27.620098 25813 consensus_queue.cc:260] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62979bde45314932a8f12c38f229e39d" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36077 } } peers { permanent_uuid: "7c7188a3e9e743058e6201eb8c6ec770" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37189 } } peers { permanent_uuid: "e1fb75c696af4d6e89fe2cd117d39573" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37193 } }
I20260923 20:20:27.622553 25813 ts_tablet_manager.cc:1428] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260923 20:20:27.628190 25814 raft_consensus.cc:348] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62979bde45314932a8f12c38f229e39d" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36077 } } peers { permanent_uuid: "7c7188a3e9e743058e6201eb8c6ec770" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37189 } } peers { permanent_uuid: "e1fb75c696af4d6e89fe2cd117d39573" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37193 } }
I20260923 20:20:27.628696 25814 raft_consensus.cc:374] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:27.628793 25814 raft_consensus.cc:729] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62979bde45314932a8f12c38f229e39d, State: Initialized, Role: FOLLOWER
I20260923 20:20:27.629456 25814 consensus_queue.cc:260] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62979bde45314932a8f12c38f229e39d" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36077 } } peers { permanent_uuid: "7c7188a3e9e743058e6201eb8c6ec770" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37189 } } peers { permanent_uuid: "e1fb75c696af4d6e89fe2cd117d39573" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37193 } }
I20260923 20:20:27.630182 25815 raft_consensus.cc:348] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62979bde45314932a8f12c38f229e39d" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36077 } } peers { permanent_uuid: "7c7188a3e9e743058e6201eb8c6ec770" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37189 } } peers { permanent_uuid: "e1fb75c696af4d6e89fe2cd117d39573" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37193 } }
I20260923 20:20:27.630604 25815 raft_consensus.cc:374] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:27.630645 25795 heartbeater.cc:503] Master 127.5.83.126:39683 was elected leader, sending a full tablet report...
I20260923 20:20:27.630708 25815 raft_consensus.cc:729] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fb75c696af4d6e89fe2cd117d39573, State: Initialized, Role: FOLLOWER
I20260923 20:20:27.631367 25814 ts_tablet_manager.cc:1428] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260923 20:20:27.631294 25815 consensus_queue.cc:260] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62979bde45314932a8f12c38f229e39d" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36077 } } peers { permanent_uuid: "7c7188a3e9e743058e6201eb8c6ec770" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37189 } } peers { permanent_uuid: "e1fb75c696af4d6e89fe2cd117d39573" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37193 } }
I20260923 20:20:27.632468 25664 heartbeater.cc:503] Master 127.5.83.126:39683 was elected leader, sending a full tablet report...
I20260923 20:20:27.633131 25815 ts_tablet_manager.cc:1428] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260923 20:20:27.712641 25819 raft_consensus.cc:482] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 20:20:27.713030 25819 raft_consensus.cc:504] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62979bde45314932a8f12c38f229e39d" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36077 } } peers { permanent_uuid: "7c7188a3e9e743058e6201eb8c6ec770" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37189 } } peers { permanent_uuid: "e1fb75c696af4d6e89fe2cd117d39573" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37193 } }
I20260923 20:20:27.714834 25819 leader_election.cc:288] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62979bde45314932a8f12c38f229e39d (127.5.83.67:36077), e1fb75c696af4d6e89fe2cd117d39573 (127.5.83.66:37193)
I20260923 20:20:27.721309 25809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.715328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:38893 (local address 127.5.83.67:36077)
0923 20:20:27.715528 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:27.715532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:27.716060 (+ 528us) server_negotiation.cc:407] Connection header received
0923 20:20:27.716284 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:27.716294 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:27.716378 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:27.716515 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:27.717122 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.717841 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.719614 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.720385 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.720533 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.720662 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:27.720935 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 20:20:27.721058 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:20:27.721365 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.715214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.67:36077 (local address 127.5.83.65:38893)
0923 20:20:27.715943 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:27.715964 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:27.716116 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:27.716607 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:27.716612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:27.716647 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:27.716971 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.716981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.717982 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.717989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.719418 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.719431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.720621 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.720628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.720777 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.720800 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 20:20:27.720908 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 20:20:27.721049 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":408,"threads_started":1}
I20260923 20:20:27.721995 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.716261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.83.65:45449 (local address 127.5.83.66:37193)
0923 20:20:27.716452 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:27.716460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:27.716883 (+ 423us) server_negotiation.cc:407] Connection header received
0923 20:20:27.717082 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:27.717086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:27.717161 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:27.717261 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 20:20:27.717829 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.718527 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.720211 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.721025 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.721173 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.721267 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:27.721683 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 20:20:27.721815 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:20:27.722062 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.716074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.66:37193 (local address 127.5.83.65:45449)
0923 20:20:27.716764 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:27.716778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 20:20:27.716931 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:20:27.717274 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:20:27.717279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:20:27.717309 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 20:20:27.717664 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.717673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.718666 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.718671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.720027 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:20:27.720039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.721221 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.721228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:20:27.721439 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.721486 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 20:20:27.721627 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 20:20:27.721763 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":350,"threads_started":1}
I20260923 20:20:27.722648 25749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720d2280c5134f3c869f60efc927ab25" candidate_uuid: "7c7188a3e9e743058e6201eb8c6ec770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62979bde45314932a8f12c38f229e39d" is_pre_election: true
I20260923 20:20:27.723166 25749 raft_consensus.cc:2454] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c7188a3e9e743058e6201eb8c6ec770 in term 0.
I20260923 20:20:27.723060 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720d2280c5134f3c869f60efc927ab25" candidate_uuid: "7c7188a3e9e743058e6201eb8c6ec770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1fb75c696af4d6e89fe2cd117d39573" is_pre_election: true
I20260923 20:20:27.723551 25618 raft_consensus.cc:2454] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c7188a3e9e743058e6201eb8c6ec770 in term 0.
I20260923 20:20:27.723846 25421 leader_election.cc:302] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62979bde45314932a8f12c38f229e39d, 7c7188a3e9e743058e6201eb8c6ec770; no voters:
I20260923 20:20:27.724310 25819 raft_consensus.cc:2781] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 20:20:27.724480 25819 raft_consensus.cc:482] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 20:20:27.724632 25819 raft_consensus.cc:3037] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:27.726989 25819 raft_consensus.cc:504] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62979bde45314932a8f12c38f229e39d" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36077 } } peers { permanent_uuid: "7c7188a3e9e743058e6201eb8c6ec770" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37189 } } peers { permanent_uuid: "e1fb75c696af4d6e89fe2cd117d39573" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37193 } }
I20260923 20:20:27.727699 25819 leader_election.cc:288] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [CANDIDATE]: Term 1 election: Requested vote from peers 62979bde45314932a8f12c38f229e39d (127.5.83.67:36077), e1fb75c696af4d6e89fe2cd117d39573 (127.5.83.66:37193)
I20260923 20:20:27.728124 25749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720d2280c5134f3c869f60efc927ab25" candidate_uuid: "7c7188a3e9e743058e6201eb8c6ec770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62979bde45314932a8f12c38f229e39d"
I20260923 20:20:27.728376 25749 raft_consensus.cc:3037] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:27.728451 25618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "720d2280c5134f3c869f60efc927ab25" candidate_uuid: "7c7188a3e9e743058e6201eb8c6ec770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1fb75c696af4d6e89fe2cd117d39573"
I20260923 20:20:27.728719 25618 raft_consensus.cc:3037] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:27.730670 25749 raft_consensus.cc:2454] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c7188a3e9e743058e6201eb8c6ec770 in term 1.
I20260923 20:20:27.730984 25618 raft_consensus.cc:2454] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c7188a3e9e743058e6201eb8c6ec770 in term 1.
I20260923 20:20:27.731034 25421 leader_election.cc:302] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62979bde45314932a8f12c38f229e39d, 7c7188a3e9e743058e6201eb8c6ec770; no voters:
I20260923 20:20:27.731323 25819 raft_consensus.cc:2781] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:27.731778 25819 raft_consensus.cc:686] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 1 LEADER]: Becoming Leader. State: Replica: 7c7188a3e9e743058e6201eb8c6ec770, State: Running, Role: LEADER
I20260923 20:20:27.732259 25819 consensus_queue.cc:237] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62979bde45314932a8f12c38f229e39d" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36077 } } peers { permanent_uuid: "7c7188a3e9e743058e6201eb8c6ec770" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37189 } } peers { permanent_uuid: "e1fb75c696af4d6e89fe2cd117d39573" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37193 } }
I20260923 20:20:27.736909 25344 catalog_manager.cc:5495] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c7188a3e9e743058e6201eb8c6ec770 (127.5.83.65). New cstate: current_term: 1 leader_uuid: "7c7188a3e9e743058e6201eb8c6ec770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62979bde45314932a8f12c38f229e39d" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c7188a3e9e743058e6201eb8c6ec770" member_type: VOTER last_known_addr { host: "127.5.83.65" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1fb75c696af4d6e89fe2cd117d39573" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37193 } health_report { overall_health: UNKNOWN } } }
W20260923 20:20:27.767120 25796 tablet.cc:2369] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:20:27.777127 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.771594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.5.83.65:37189)
0923 20:20:27.771799 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:27.771806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:27.771825 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:20:27.772080 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:27.772084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:27.772173 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:27.772324 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:20:27.772759 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.773469 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.774889 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.775310 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.775445 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.775956 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:20:27.776500 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:20:27.776695 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:27.776758 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 20:20:27.776840 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:20:27.785458 25749 raft_consensus.cc:1261] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d [term 1 FOLLOWER]: Refusing update from remote peer 7c7188a3e9e743058e6201eb8c6ec770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:20:27.785462 25618 raft_consensus.cc:1261] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573 [term 1 FOLLOWER]: Refusing update from remote peer 7c7188a3e9e743058e6201eb8c6ec770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:20:27.786428 25824 consensus_queue.cc:1035] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62979bde45314932a8f12c38f229e39d" member_type: VOTER last_known_addr { host: "127.5.83.67" port: 36077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:20:27.786845 25819 consensus_queue.cc:1035] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1fb75c696af4d6e89fe2cd117d39573" member_type: VOTER last_known_addr { host: "127.5.83.66" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 20:20:27.795239 25665 tablet.cc:2369] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:20:27.801941 25829 mvcc.cc:205] Tried to move back new op lower bound from 7332638014599077888 to 7332638014395580416. Current Snapshot: MvccSnapshot[applied={T|T < 7332638014599077888}]
I20260923 20:20:27.815531 25344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:20:27.815811 25344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 20:20:27.819761 25344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ab15257d2a941e8b7c6107f7d8d9c0d: Sending DeleteTablet for 3 replicas of tablet 720d2280c5134f3c869f60efc927ab25
I20260923 20:20:27.820979 25465 tablet_service.cc:1514] Processing DeleteTablet for tablet 720d2280c5134f3c869f60efc927ab25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:27 UTC) from {username='slave'} at 127.0.0.1:37034
I20260923 20:20:27.821053 25598 tablet_service.cc:1514] Processing DeleteTablet for tablet 720d2280c5134f3c869f60efc927ab25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:27 UTC) from {username='slave'} at 127.0.0.1:34072
I20260923 20:20:27.821056 25729 tablet_service.cc:1514] Processing DeleteTablet for tablet 720d2280c5134f3c869f60efc927ab25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 20:20:27 UTC) from {username='slave'} at 127.0.0.1:38604
I20260923 20:20:27.822702 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 25403
I20260923 20:20:27.833431 25838 tablet_replica.cc:330] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573: stopping tablet replica
I20260923 20:20:27.834162 25837 tablet_replica.cc:330] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770: stopping tablet replica
I20260923 20:20:27.835129 25839 tablet_replica.cc:330] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d: stopping tablet replica
I20260923 20:20:27.835418 25838 raft_consensus.cc:2229] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:20:27.835625 25839 raft_consensus.cc:2229] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 20:20:27.835907 25837 raft_consensus.cc:2229] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 1 LEADER]: Raft consensus shutting down.
I20260923 20:20:27.836212 25839 raft_consensus.cc:2258] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:27.837158 25837 raft_consensus.cc:2258] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:27.837661 25838 raft_consensus.cc:2258] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 20:20:27.838546 25839 ts_tablet_manager.cc:1905] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:27.840184 25838 ts_tablet_manager.cc:1905] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:27.842016 25837 ts_tablet_manager.cc:1905] T 720d2280c5134f3c869f60efc927ab25 P 7c7188a3e9e743058e6201eb8c6ec770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 20:20:27.845793 25839 ts_tablet_manager.cc:1918] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:27.845966 25839 log.cc:1196] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-2/wal/wals/720d2280c5134f3c869f60efc927ab25
I20260923 20:20:27.843628 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.825121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.5.83.65:37189)
0923 20:20:27.825346 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:27.825355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:27.834806 (+ 9451us) server_negotiation.cc:407] Connection header received
0923 20:20:27.834885 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:27.834891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:27.834997 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:27.835105 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:27.836140 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.837347 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.839463 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.840117 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.840226 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.841832 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:27.841843 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:27.841859 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:27.842620 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:27.842689 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:27.842716 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:27.842816 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:27.843099 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 20:20:27.843222 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:20:27.846661 25525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:27.847328 25839 ts_tablet_manager.cc:1939] T 720d2280c5134f3c869f60efc927ab25 P 62979bde45314932a8f12c38f229e39d: Deleting consensus metadata
I20260923 20:20:27.848930 25838 ts_tablet_manager.cc:1918] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 20:20:27.849269 25838 log.cc:1196] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573: Deleting WAL directory at /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790194506409421-5453-0/minicluster-data/ts-1/wal/wals/720d2280c5134f3c869f60efc927ab25
I20260923 20:20:27.849378 25331 catalog_manager.cc:4841] TS 62979bde45314932a8f12c38f229e39d (127.5.83.67:36077): tablet 720d2280c5134f3c869f60efc927ab25 (table test-table [id=47821c4aa12b48bd9508487ef97b937d]) successfully deleted
I20260923 20:20:27.849886 25838 ts_tablet_manager.cc:1939] T 720d2280c5134f3c869f60efc927ab25 P e1fb75c696af4d6e89fe2cd117d39573: Deleting consensus metadata
I20260923 20:20:27.851593 25331 catalog_manager.cc:4841] TS e1fb75c696af4d6e89fe2cd117d39573 (127.5.83.66:37193): tablet 720d2280c5134f3c869f60efc927ab25 (table test-table [id=47821c4aa12b48bd9508487ef97b937d]) successfully deleted
I20260923 20:20:27.943297 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 25403
W20260923 20:20:27.964627 25330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.83.65:37189 (error 108)
I20260923 20:20:27.964793 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 25535
I20260923 20:20:27.966645 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.966167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37189 (local address 127.0.0.1:37068)
0923 20:20:27.966354 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:27.966484 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 20:20:27.966890 25330 catalog_manager.cc:4584] TS 7c7188a3e9e743058e6201eb8c6ec770 (127.5.83.65:37189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 720d2280c5134f3c869f60efc927ab25: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37189: connect: Connection refused (error 111)
I20260923 20:20:27.971961 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:27.965706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.5.83.66:37193)
0923 20:20:27.965912 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:27.965918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:27.965939 (+ 21us) server_negotiation.cc:407] Connection header received
0923 20:20:27.966060 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:27.966070 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:27.966159 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:27.966274 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:27.966707 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.967596 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.969031 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:27.969435 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:27.969539 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:27.970585 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:27.970592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:27.970606 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:27.971225 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:27.971270 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:27.971288 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:27.971379 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:27.971493 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 20:20:27.971587 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 20:20:27.972676 25659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:28.020851 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:28.020312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37189 (local address 127.0.0.1:37078)
0923 20:20:28.020544 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:28.020654 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 20:20:28.021951 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:28.021555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37189 (local address 127.0.0.1:37088)
0923 20:20:28.021734 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:28.021811 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260923 20:20:28.061959 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 25535
I20260923 20:20:28.080189 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 25667
I20260923 20:20:28.082422 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:28.081915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37189 (local address 127.0.0.1:37104)
0923 20:20:28.082109 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:28.082223 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 20:20:28.083462 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:28.083087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37189 (local address 127.0.0.1:37118)
0923 20:20:28.083231 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:28.083313 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 20:20:28.087479 25809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:28.080994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.5.83.67:36077)
0923 20:20:28.081208 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:28.081215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:28.081303 (+ 88us) server_negotiation.cc:407] Connection header received
0923 20:20:28.081536 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:28.081542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:28.081641 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:28.081753 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:28.082288 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:28.083316 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:28.084715 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:28.085105 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:28.085207 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:28.086298 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:28.086306 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:28.086321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:28.086905 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:28.086951 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:28.086970 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:28.087055 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:28.087165 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 20:20:28.087258 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 20:20:28.088130 25790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:28.164040 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:28.163527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37189 (local address 127.0.0.1:37134)
0923 20:20:28.163752 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:28.163868 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 20:20:28.165269 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:28.164894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.65:37189 (local address 127.0.0.1:37140)
0923 20:20:28.165077 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 20:20:28.165156 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.65:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 20:20:28.182684 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 25667
I20260923 20:20:28.199855 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 25313
I20260923 20:20:28.201015 25374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:28.311028 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 25313
2026-09-23T20:20:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2821 ms)
[----------] 6 tests from EncryptionPolicyTest (22365 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:28 dist-test-slave-vl24 krb5kdc[25848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:28 dist-test-slave-vl24 krb5kdc[25848](info): set up 2 sockets
Sep 23 20:20:28 dist-test-slave-vl24 krb5kdc[25848](info): commencing operation
krb5kdc: starting...
W20260923 20:20:29.454756 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.005s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:29 dist-test-slave-vl24 krb5kdc[25848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:20:29.747992 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:30.089563 25861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:30.089941 25861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:30.090121 25861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:30.097513 25861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:30.097684 25861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:30.097797 25861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:30.097908 25861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:30.097990 25861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:30.107918 25861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:30.109393 25861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:30.110718 25861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:30.123136 25866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:30.123188 25867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:30.123101 25869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:30.124421 25861 server_base.cc:1034] running on GCE node
I20260923 20:20:30.125223 25861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:30.126606 25861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:30.127825 25861 hybrid_clock.cc:648] HybridClock initialized: now 1790194830127777 us; error 32 us; skew 500 ppm
I20260923 20:20:30.128137 25861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:30 dist-test-slave-vl24 krb5kdc[25848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:30.131544 25861 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:20:30.133095 25861 webserver.cc:460] Webserver started at http://127.0.0.1:36829/ using document root <none> and password file <none>
I20260923 20:20:30.133818 25861 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:30.133970 25861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:30.134296 25861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:30.137499 25861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "e82a4ca4908342a3a1219dfdf05575cc"
format_stamp: "Formatted at 2026-09-23 20:20:30 on dist-test-slave-vl24"
server_key: "425163b5776e3466d3928c3581d14876"
server_key_iv: "d6b16df878dc4fdadf5b871d4a2d2b8e"
server_key_version: "encryptionkey@0"
I20260923 20:20:30.138304 25861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "e82a4ca4908342a3a1219dfdf05575cc"
format_stamp: "Formatted at 2026-09-23 20:20:30 on dist-test-slave-vl24"
server_key: "425163b5776e3466d3928c3581d14876"
server_key_iv: "d6b16df878dc4fdadf5b871d4a2d2b8e"
server_key_version: "encryptionkey@0"
I20260923 20:20:30.143560 25861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:20:30.147483 25876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:30.148862 25861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:20:30.149056 25861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "e82a4ca4908342a3a1219dfdf05575cc"
format_stamp: "Formatted at 2026-09-23 20:20:30 on dist-test-slave-vl24"
server_key: "425163b5776e3466d3928c3581d14876"
server_key_iv: "d6b16df878dc4fdadf5b871d4a2d2b8e"
server_key_version: "encryptionkey@0"
I20260923 20:20:30.149272 25861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:30.174575 25861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:30.178912 25861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:30.179215 25861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:30.199011 25861 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41469
I20260923 20:20:30.199033 25928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41469 every 8 connection(s)
I20260923 20:20:30.200438 25861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:30.204145 25929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:30.206118 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 25861
I20260923 20:20:30.206327 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:30.206997 5453 external_mini_cluster.cc:1428] Setting key 687b499f5d441e4cf9b8a61fabfb625c
I20260923 20:20:30.216127 25929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc: Bootstrap starting.
I20260923 20:20:30.219628 25929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:30.220870 25929 log.cc:824] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:20:30 dist-test-slave-vl24 krb5kdc[25848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:20:30.224460 25929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc: No bootstrap required, opened a new log
I20260923 20:20:30.231915 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:30.209122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51823 (local address 127.0.0.1:41469)
0923 20:20:30.209928 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:30.209940 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:30.209986 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:20:30.211660 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:30.211681 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:30.212074 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:30.212459 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:30.214071 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:30.215379 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:30.216770 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:30.217428 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:30.217629 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:30.222330 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:30.222356 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:30.222376 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:30.222418 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:30.226641 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:30.227611 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:30.227621 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:30.227630 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:30.227880 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:30.228360 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:30.228366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:30.228370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:30.229166 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:30.229551 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:30.229933 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 20:20:30.230338 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":355,"threads_started":1}
I20260923 20:20:30.235893 25929 raft_consensus.cc:348] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82a4ca4908342a3a1219dfdf05575cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41469 } }
I20260923 20:20:30.236323 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:30.236430 25929 raft_consensus.cc:374] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:30.236621 25929 raft_consensus.cc:729] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82a4ca4908342a3a1219dfdf05575cc, State: Initialized, Role: FOLLOWER
I20260923 20:20:30.237243 25929 consensus_queue.cc:260] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82a4ca4908342a3a1219dfdf05575cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41469 } }
I20260923 20:20:30.237509 25929 raft_consensus.cc:388] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:30.237666 25929 raft_consensus.cc:482] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:30.237871 25929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:30.240365 25929 raft_consensus.cc:504] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82a4ca4908342a3a1219dfdf05575cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41469 } }
I20260923 20:20:30.240926 25929 leader_election.cc:302] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e82a4ca4908342a3a1219dfdf05575cc; no voters:
I20260923 20:20:30.241654 25929 leader_election.cc:288] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:30.241773 25938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:30.243083 25938 raft_consensus.cc:686] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [term 1 LEADER]: Becoming Leader. State: Replica: e82a4ca4908342a3a1219dfdf05575cc, State: Running, Role: LEADER
I20260923 20:20:30.243892 25938 consensus_queue.cc:237] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82a4ca4908342a3a1219dfdf05575cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41469 } }
I20260923 20:20:30.245385 25929 sys_catalog.cc:564] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:30.247031 25941 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e82a4ca4908342a3a1219dfdf05575cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82a4ca4908342a3a1219dfdf05575cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41469 } } }
I20260923 20:20:30.247682 25941 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:30.248126 25939 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e82a4ca4908342a3a1219dfdf05575cc. Latest consensus state: current_term: 1 leader_uuid: "e82a4ca4908342a3a1219dfdf05575cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82a4ca4908342a3a1219dfdf05575cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41469 } } }
I20260923 20:20:30.248957 25939 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:30.253720 25944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:30.261471 25944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:30.262722 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:30.241864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40623 (local address 127.0.0.1:41469)
0923 20:20:30.242048 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:30.242058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:30.242638 (+ 580us) server_negotiation.cc:407] Connection header received
0923 20:20:30.242850 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:30.242858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:30.242956 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:30.243076 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:30.244936 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:30.246002 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:30.247885 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:30.248632 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:30.248758 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:30.253027 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:30.253036 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:30.253046 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:30.253106 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:30.257617 (+ 4511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:30.258779 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:30.258785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:30.258788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:30.258976 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:30.259685 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:30.259694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:30.259700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:30.260198 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:30.260583 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:30.261870 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0923 20:20:30.262153 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 20:20:30.270411 25956 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:20:30.273483 25944 catalog_manager.cc:1347] Generated new cluster ID: d33292bb32124b4f91cca4f355b191c1
I20260923 20:20:30.273633 25944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:30.285076 25944 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:30.286981 25944 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:30.306149 25944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e82a4ca4908342a3a1219dfdf05575cc: Generated new TSK 0
I20260923 20:20:30.307343 25944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:30.322835 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 25861
I20260923 20:20:30.324359 25924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:30.418258 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 25861
W20260923 20:20:30.437233 25846 connection.cc:535] client connection to 127.0.0.1:41469 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41469): Connection reset by peer (error 104)
2026-09-23T20:20:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2096 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:30 dist-test-slave-vl24 krb5kdc[25961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:30 dist-test-slave-vl24 krb5kdc[25961](info): set up 2 sockets
Sep 23 20:20:30 dist-test-slave-vl24 krb5kdc[25961](info): commencing operation
krb5kdc: starting...
W20260923 20:20:31.546229 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:31 dist-test-slave-vl24 krb5kdc[25961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:20:31.828173 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:32.187448 25974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:32.187879 25974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:32.188053 25974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:32.196269 25974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:32.196403 25974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:32.196552 25974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:32.196650 25974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:32.196735 25974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:32.206980 25974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35907
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:32.208317 25974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:32.209712 25974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:32.221995 25980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:32.222090 25979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:32.222088 25982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:32.222944 25974 server_base.cc:1034] running on GCE node
I20260923 20:20:32.223786 25974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:32.225307 25974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:32.226516 25974 hybrid_clock.cc:648] HybridClock initialized: now 1790194832226481 us; error 25 us; skew 500 ppm
I20260923 20:20:32.226856 25974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:32 dist-test-slave-vl24 krb5kdc[25961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:32.229884 25974 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:20:32.231348 25974 webserver.cc:460] Webserver started at http://127.0.0.1:46743/ using document root <none> and password file <none>
I20260923 20:20:32.232040 25974 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:32.232191 25974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:32.232498 25974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:32.235667 25974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "6e03589be5e1426bb2a086719c3c8b4a"
format_stamp: "Formatted at 2026-09-23 20:20:32 on dist-test-slave-vl24"
server_key: "6d0cb4846658ef0a8163a2148a3eb453"
server_key_iv: "bd8b53e1055438b1f438b6c6cfd9d61f"
server_key_version: "encryptionkey@0"
I20260923 20:20:32.236459 25974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "6e03589be5e1426bb2a086719c3c8b4a"
format_stamp: "Formatted at 2026-09-23 20:20:32 on dist-test-slave-vl24"
server_key: "6d0cb4846658ef0a8163a2148a3eb453"
server_key_iv: "bd8b53e1055438b1f438b6c6cfd9d61f"
server_key_version: "encryptionkey@0"
I20260923 20:20:32.242089 25974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:20:32.245692 25989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:32.247092 25974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:20:32.247287 25974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "6e03589be5e1426bb2a086719c3c8b4a"
format_stamp: "Formatted at 2026-09-23 20:20:32 on dist-test-slave-vl24"
server_key: "6d0cb4846658ef0a8163a2148a3eb453"
server_key_iv: "bd8b53e1055438b1f438b6c6cfd9d61f"
server_key_version: "encryptionkey@0"
I20260923 20:20:32.247504 25974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:32.266686 25974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:32.271013 25974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:32.271317 25974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:32.290706 25974 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35907
I20260923 20:20:32.290737 26040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35907 every 8 connection(s)
I20260923 20:20:32.292162 25974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:32.296031 26041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:32.296638 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 25974
I20260923 20:20:32.296857 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:32.297307 5453 external_mini_cluster.cc:1428] Setting key 47269eae4c72c520ab49883ea0149e79
I20260923 20:20:32.311635 26041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a: Bootstrap starting.
I20260923 20:20:32.313848 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:32.299231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55337 (local address 127.0.0.1:35907)
0923 20:20:32.300053 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:32.300067 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:32.300117 (+ 50us) server_negotiation.cc:407] Connection header received
0923 20:20:32.302935 (+ 2818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:32.302969 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:32.303547 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:32.304038 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:32.304611 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:32.306505 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:32.308021 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:32.309231 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:32.309422 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:32.310801 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:32.310812 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:32.310834 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:32.311605 (+ 771us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:32.311718 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:32.311754 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:32.311917 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:32.312225 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 20:20:32.312417 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":342,"threads_started":1}
I20260923 20:20:32.315903 26041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:32.317251 26041 log.cc:824] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:32.318349 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:32.320932 26041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a: No bootstrap required, opened a new log
I20260923 20:20:32.331986 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:32.324083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59539 (local address 127.0.0.1:35907)
0923 20:20:32.324443 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:32.324452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:32.324827 (+ 375us) server_negotiation.cc:407] Connection header received
0923 20:20:32.324976 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:32.324980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:32.325048 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:32.325145 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:32.325817 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:32.326623 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:32.328857 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:32.329393 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:32.329519 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:32.331005 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:32.331012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:32.331016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:32.331045 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:32.331100 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:32.331118 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:32.331359 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:32.331535 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 20:20:32.331587 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 20:20:32.332103 26041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e03589be5e1426bb2a086719c3c8b4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35907 } }
I20260923 20:20:32.332669 26041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:32.332803 26041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e03589be5e1426bb2a086719c3c8b4a, State: Initialized, Role: FOLLOWER
I20260923 20:20:32.333528 26041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e03589be5e1426bb2a086719c3c8b4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35907 } }
I20260923 20:20:32.333824 26041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:32.333978 26041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:32.334172 26041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:32.336400 26041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e03589be5e1426bb2a086719c3c8b4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35907 } }
I20260923 20:20:32.337087 26041 leader_election.cc:302] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e03589be5e1426bb2a086719c3c8b4a; no voters:
I20260923 20:20:32.337706 26041 leader_election.cc:288] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:32.337834 26051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:32.338593 26051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [term 1 LEADER]: Becoming Leader. State: Replica: 6e03589be5e1426bb2a086719c3c8b4a, State: Running, Role: LEADER
I20260923 20:20:32.339179 26051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e03589be5e1426bb2a086719c3c8b4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35907 } }
I20260923 20:20:32.339883 26041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:32.342403 26052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e03589be5e1426bb2a086719c3c8b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e03589be5e1426bb2a086719c3c8b4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35907 } } }
I20260923 20:20:32.342859 26052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:32.342375 26053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e03589be5e1426bb2a086719c3c8b4a. Latest consensus state: current_term: 1 leader_uuid: "6e03589be5e1426bb2a086719c3c8b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e03589be5e1426bb2a086719c3c8b4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35907 } } }
I20260923 20:20:32.344064 26053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a [sys.catalog]: This master's current role is: LEADER
W20260923 20:20:32.351315 26067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:20:32.351430 26067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:20:32.351624 26060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:32.353466 26060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:32.362064 26060 catalog_manager.cc:1347] Generated new cluster ID: a1881d5c005e4497b54441f76be5f3c3
I20260923 20:20:32.362183 26060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:32.375236 26060 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:32.376639 26060 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:32.388139 26060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e03589be5e1426bb2a086719c3c8b4a: Generated new TSK 0
I20260923 20:20:32.389163 26060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:32.401682 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 25974
W20260923 20:20:32.402907 26036 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55337
W20260923 20:20:32.403735 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 25974: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:20:32.403913 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 25974
W20260923 20:20:32.422704 25959 connection.cc:535] client connection to 127.0.0.1:35907 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35907): Connection reset by peer (error 104)
2026-09-23T20:20:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:32 dist-test-slave-vl24 krb5kdc[26072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:32 dist-test-slave-vl24 krb5kdc[26072](info): set up 2 sockets
Sep 23 20:20:32 dist-test-slave-vl24 krb5kdc[26072](info): commencing operation
krb5kdc: starting...
W20260923 20:20:33.514916 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:33 dist-test-slave-vl24 krb5kdc[26072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:20:33.799271 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:34.142529 26085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:34.142824 26085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:34.142982 26085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:34.150385 26085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:34.150508 26085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:34.150607 26085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:34.150718 26085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:34.150810 26085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:34.161181 26085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:34.162516 26085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:34.163858 26085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:34.175788 26093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:34.175792 26090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:34.175875 26091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:34.176841 26085 server_base.cc:1034] running on GCE node
I20260923 20:20:34.177654 26085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:34.179042 26085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:34.180289 26085 hybrid_clock.cc:648] HybridClock initialized: now 1790194834180232 us; error 30 us; skew 500 ppm
I20260923 20:20:34.180831 26085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:34 dist-test-slave-vl24 krb5kdc[26072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:34.187634 26085 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:20:34.189481 26085 webserver.cc:460] Webserver started at http://127.0.0.1:37549/ using document root <none> and password file <none>
I20260923 20:20:34.190260 26085 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:34.190423 26085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:34.190783 26085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:34.194214 26085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "c1a479b0b8ad4b95931f80b8b3aa1b90"
format_stamp: "Formatted at 2026-09-23 20:20:34 on dist-test-slave-vl24"
server_key: "b08e6ec027419329b6dae7303357bbc2"
server_key_iv: "31f87be649f231c32fa1c79383ddc380"
server_key_version: "encryptionkey@0"
I20260923 20:20:34.195142 26085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "c1a479b0b8ad4b95931f80b8b3aa1b90"
format_stamp: "Formatted at 2026-09-23 20:20:34 on dist-test-slave-vl24"
server_key: "b08e6ec027419329b6dae7303357bbc2"
server_key_iv: "31f87be649f231c32fa1c79383ddc380"
server_key_version: "encryptionkey@0"
I20260923 20:20:34.200955 26085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:20:34.204631 26100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:34.206110 26085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:20:34.206323 26085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "c1a479b0b8ad4b95931f80b8b3aa1b90"
format_stamp: "Formatted at 2026-09-23 20:20:34 on dist-test-slave-vl24"
server_key: "b08e6ec027419329b6dae7303357bbc2"
server_key_iv: "31f87be649f231c32fa1c79383ddc380"
server_key_version: "encryptionkey@0"
I20260923 20:20:34.206544 26085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:34.212616 26085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:34.217208 26085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:34.217516 26085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:34.236392 26085 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38549
I20260923 20:20:34.236431 26151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38549 every 8 connection(s)
I20260923 20:20:34.237861 26085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:34.242103 26152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:34.247498 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 26085
I20260923 20:20:34.247691 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:34.248134 5453 external_mini_cluster.cc:1428] Setting key 9aa444ea0d6bb9039cf0cd1a197d91e8
I20260923 20:20:34.254106 26152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90: Bootstrap starting.
I20260923 20:20:34.257246 26155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:34.250014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36443 (local address 127.0.0.1:38549)
0923 20:20:34.250889 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:34.250905 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:34.250964 (+ 59us) server_negotiation.cc:407] Connection header received
0923 20:20:34.253371 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:34.253400 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:34.253837 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:34.254267 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:34.255237 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:34.255249 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:34.255264 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:34.255299 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:34.255365 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:34.255393 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:34.255613 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:34.255751 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 20:20:34.255978 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":355,"threads_started":1}
I20260923 20:20:34.258090 26152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:34.259182 26152 log.cc:824] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:34.261274 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:34.262442 26152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90: No bootstrap required, opened a new log
I20260923 20:20:34.268079 26155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:34.265497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36283 (local address 127.0.0.1:38549)
0923 20:20:34.265730 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:34.265743 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:34.266135 (+ 392us) server_negotiation.cc:407] Connection header received
0923 20:20:34.266297 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:34.266300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:34.266349 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:34.266432 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:34.267389 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:34.267392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:34.267395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:34.267411 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:34.267457 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:34.267474 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:34.267564 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:34.267702 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 20:20:34.267750 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:20:34.271471 26152 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a479b0b8ad4b95931f80b8b3aa1b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38549 } }
I20260923 20:20:34.271869 26152 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:34.271991 26152 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1a479b0b8ad4b95931f80b8b3aa1b90, State: Initialized, Role: FOLLOWER
I20260923 20:20:34.272713 26152 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a479b0b8ad4b95931f80b8b3aa1b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38549 } }
I20260923 20:20:34.272970 26152 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:34.273119 26152 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:34.273313 26152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:34.275652 26152 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a479b0b8ad4b95931f80b8b3aa1b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38549 } }
I20260923 20:20:34.276192 26152 leader_election.cc:302] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1a479b0b8ad4b95931f80b8b3aa1b90; no voters:
I20260923 20:20:34.276844 26152 leader_election.cc:288] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:34.276973 26162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:34.277737 26162 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [term 1 LEADER]: Becoming Leader. State: Replica: c1a479b0b8ad4b95931f80b8b3aa1b90, State: Running, Role: LEADER
I20260923 20:20:34.278231 26162 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a479b0b8ad4b95931f80b8b3aa1b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38549 } }
I20260923 20:20:34.278978 26152 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:34.281225 26164 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1a479b0b8ad4b95931f80b8b3aa1b90. Latest consensus state: current_term: 1 leader_uuid: "c1a479b0b8ad4b95931f80b8b3aa1b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a479b0b8ad4b95931f80b8b3aa1b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38549 } } }
I20260923 20:20:34.281734 26164 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:34.285897 26163 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1a479b0b8ad4b95931f80b8b3aa1b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1a479b0b8ad4b95931f80b8b3aa1b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38549 } } }
I20260923 20:20:34.286543 26163 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:34.287400 26171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:34.292634 26171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:34.302223 26171 catalog_manager.cc:1347] Generated new cluster ID: 316a67f809b84c11b5f6a2fa422da86f
I20260923 20:20:34.302356 26171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:34.335484 26171 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:34.335952 26171 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:34.349735 26171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1a479b0b8ad4b95931f80b8b3aa1b90: Generated new TSK 0
I20260923 20:20:34.350835 26171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:34.362658 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26085
W20260923 20:20:34.364205 26147 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36443
W20260923 20:20:34.365001 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26085: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:20:34.365160 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 26085
2026-09-23T20:20:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:34 dist-test-slave-vl24 krb5kdc[26183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:34 dist-test-slave-vl24 krb5kdc[26183](info): set up 2 sockets
Sep 23 20:20:34 dist-test-slave-vl24 krb5kdc[26183](info): commencing operation
krb5kdc: starting...
W20260923 20:20:35.498466 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:35 dist-test-slave-vl24 krb5kdc[26183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:20:35.788074 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:36.127040 26196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:36.127333 26196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:36.127486 26196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:36.135284 26196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:36.135398 26196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:36.135506 26196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:36.135612 26196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:36.135675 26196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:36.146071 26196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:36.147466 26196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:36.148823 26196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:36.160902 26201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:36.160995 26202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:36.160972 26204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:36.162169 26196 server_base.cc:1034] running on GCE node
I20260923 20:20:36.162992 26196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:36.164515 26196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:36.165789 26196 hybrid_clock.cc:648] HybridClock initialized: now 1790194836165737 us; error 47 us; skew 500 ppm
I20260923 20:20:36.166174 26196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:36 dist-test-slave-vl24 krb5kdc[26183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:36.169407 26196 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:20:36.171038 26196 webserver.cc:460] Webserver started at http://127.0.0.1:39883/ using document root <none> and password file <none>
I20260923 20:20:36.171749 26196 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:36.171914 26196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:36.172223 26196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:36.175302 26196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "8df1931e39f44356acf31e634b4d1fe1"
format_stamp: "Formatted at 2026-09-23 20:20:36 on dist-test-slave-vl24"
server_key: "6867438dc124e4acc7b334732f44ebad"
server_key_iv: "13146c8dcfde86973968c3e4062ac264"
server_key_version: "encryptionkey@0"
I20260923 20:20:36.176082 26196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "8df1931e39f44356acf31e634b4d1fe1"
format_stamp: "Formatted at 2026-09-23 20:20:36 on dist-test-slave-vl24"
server_key: "6867438dc124e4acc7b334732f44ebad"
server_key_iv: "13146c8dcfde86973968c3e4062ac264"
server_key_version: "encryptionkey@0"
I20260923 20:20:36.181581 26196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 20:20:36.185581 26211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:36.186978 26196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 20:20:36.187172 26196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "8df1931e39f44356acf31e634b4d1fe1"
format_stamp: "Formatted at 2026-09-23 20:20:36 on dist-test-slave-vl24"
server_key: "6867438dc124e4acc7b334732f44ebad"
server_key_iv: "13146c8dcfde86973968c3e4062ac264"
server_key_version: "encryptionkey@0"
I20260923 20:20:36.187438 26196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:36.202626 26196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:36.207154 26196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:36.207402 26196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:36.227223 26196 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34245
I20260923 20:20:36.227254 26263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34245 every 8 connection(s)
I20260923 20:20:36.228639 26196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:36.232299 26264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:36.236737 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 26196
I20260923 20:20:36.236939 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:36.237490 5453 external_mini_cluster.cc:1428] Setting key 424d69a7eb0ece86ed991e59056ec187
I20260923 20:20:36.245229 26264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1: Bootstrap starting.
I20260923 20:20:36.248363 26264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:36.249433 26264 log.cc:824] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:20:36 dist-test-slave-vl24 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:20:36.252699 26264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1: No bootstrap required, opened a new log
I20260923 20:20:36.260797 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:36.239348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48091 (local address 127.0.0.1:34245)
0923 20:20:36.240086 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:36.240097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:36.240136 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:20:36.241830 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:36.241853 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:36.242248 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:36.242638 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:36.244058 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:36.245509 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:36.246817 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:36.247488 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:36.247687 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:36.252457 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:36.252466 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:36.252481 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:36.252521 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:36.256070 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:36.256999 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:36.257017 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:36.257023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:36.257227 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:36.257655 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:36.257661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:36.257665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:36.258289 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:36.258649 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:36.259173 (+ 524us) server_negotiation.cc:299] Negotiation successful
0923 20:20:36.259632 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":325,"threads_started":1}
I20260923 20:20:36.261046 26264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df1931e39f44356acf31e634b4d1fe1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } }
I20260923 20:20:36.261476 26264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:36.261624 26264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df1931e39f44356acf31e634b4d1fe1, State: Initialized, Role: FOLLOWER
I20260923 20:20:36.262190 26264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df1931e39f44356acf31e634b4d1fe1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } }
I20260923 20:20:36.262456 26264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:36.262614 26264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:36.262799 26264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:36.265195 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:36.265631 26264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df1931e39f44356acf31e634b4d1fe1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } }
I20260923 20:20:36.266126 26264 leader_election.cc:302] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8df1931e39f44356acf31e634b4d1fe1; no voters:
I20260923 20:20:36.266780 26264 leader_election.cc:288] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:36.266901 26269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:36.268250 26269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [term 1 LEADER]: Becoming Leader. State: Replica: 8df1931e39f44356acf31e634b4d1fe1, State: Running, Role: LEADER
I20260923 20:20:36.268952 26269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df1931e39f44356acf31e634b4d1fe1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } }
I20260923 20:20:36.269275 26264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:36.271996 26271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8df1931e39f44356acf31e634b4d1fe1. Latest consensus state: current_term: 1 leader_uuid: "8df1931e39f44356acf31e634b4d1fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df1931e39f44356acf31e634b4d1fe1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } } }
I20260923 20:20:36.272663 26271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:36.272441 26270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8df1931e39f44356acf31e634b4d1fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df1931e39f44356acf31e634b4d1fe1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34245 } } }
I20260923 20:20:36.272852 26270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:36.280915 26280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:36.288002 26280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:36.301911 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:36.281908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44757 (local address 127.0.0.1:34245)
0923 20:20:36.282100 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:36.282109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:36.282695 (+ 586us) server_negotiation.cc:407] Connection header received
0923 20:20:36.282903 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:36.282908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:36.282981 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:36.283087 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:36.285097 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:36.286297 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:36.288410 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:36.289170 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:36.289290 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:36.293150 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:36.293157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:36.293163 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:36.293202 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:36.297111 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:36.297954 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:36.297966 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:36.297973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:36.298156 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:36.298700 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:36.298707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:36.298713 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:36.299277 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:36.299567 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:36.301032 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0923 20:20:36.301353 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:20:36.311937 26280 catalog_manager.cc:1347] Generated new cluster ID: 32d6d1bc027c48ea93a6a90fa31e002f
I20260923 20:20:36.312078 26280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:36.320935 26280 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:36.322854 26280 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:36.333488 26280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8df1931e39f44356acf31e634b4d1fe1: Generated new TSK 0
I20260923 20:20:36.334420 26280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:36.345989 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26196
I20260923 20:20:36.347386 26259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:36.438647 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 26196
2026-09-23T20:20:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2073 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:36 dist-test-slave-vl24 krb5kdc[26296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:36 dist-test-slave-vl24 krb5kdc[26296](info): set up 2 sockets
Sep 23 20:20:36 dist-test-slave-vl24 krb5kdc[26296](info): commencing operation
krb5kdc: starting...
W20260923 20:20:37.570329 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:37 dist-test-slave-vl24 krb5kdc[26296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:20:37.859678 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:38.204352 26309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:38.204773 26309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:38.204896 26309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:38.212276 26309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:38.212394 26309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:38.212441 26309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:38.212504 26309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:38.212618 26309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:38.222932 26309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:38.224195 26309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:38.225500 26309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:38.237738 26315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:38.237759 26314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:38.237761 26317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:38.238698 26309 server_base.cc:1034] running on GCE node
I20260923 20:20:38.239581 26309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:38.241098 26309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:38.242333 26309 hybrid_clock.cc:648] HybridClock initialized: now 1790194838242288 us; error 39 us; skew 500 ppm
I20260923 20:20:38.242695 26309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:38 dist-test-slave-vl24 krb5kdc[26296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:38.245932 26309 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:20:38.247486 26309 webserver.cc:460] Webserver started at http://127.0.0.1:35181/ using document root <none> and password file <none>
I20260923 20:20:38.248199 26309 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:38.248348 26309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:38.248761 26309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:38.252370 26309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "17d09d0942844146963fe7c18defffcf"
format_stamp: "Formatted at 2026-09-23 20:20:38 on dist-test-slave-vl24"
server_key: "3aff5c9512519c7f395ceabe0a460788"
server_key_iv: "102fb270254299c86c5e030fbb728a5f"
server_key_version: "encryptionkey@0"
I20260923 20:20:38.253242 26309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "17d09d0942844146963fe7c18defffcf"
format_stamp: "Formatted at 2026-09-23 20:20:38 on dist-test-slave-vl24"
server_key: "3aff5c9512519c7f395ceabe0a460788"
server_key_iv: "102fb270254299c86c5e030fbb728a5f"
server_key_version: "encryptionkey@0"
I20260923 20:20:38.258584 26309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:20:38.262109 26324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:38.263487 26309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 20:20:38.263671 26309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "17d09d0942844146963fe7c18defffcf"
format_stamp: "Formatted at 2026-09-23 20:20:38 on dist-test-slave-vl24"
server_key: "3aff5c9512519c7f395ceabe0a460788"
server_key_iv: "102fb270254299c86c5e030fbb728a5f"
server_key_version: "encryptionkey@0"
I20260923 20:20:38.263870 26309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:38.286185 26309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:38.290455 26309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:38.290758 26309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:38.310472 26309 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46461
I20260923 20:20:38.310532 26375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46461 every 8 connection(s)
I20260923 20:20:38.311951 26309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:38.315725 26376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:38.318693 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 26309
I20260923 20:20:38.318884 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:38.319397 5453 external_mini_cluster.cc:1428] Setting key 10d576bf387bb6551376c094206c2da2
I20260923 20:20:38.328088 26376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf: Bootstrap starting.
I20260923 20:20:38.331881 26376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:38.333122 26376 log.cc:824] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:38.333480 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:38.321422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46035 (local address 127.0.0.1:46461)
0923 20:20:38.322275 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:38.322289 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:38.322332 (+ 43us) server_negotiation.cc:407] Connection header received
0923 20:20:38.324015 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:38.324040 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:38.324472 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:38.324888 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:38.325551 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:38.326896 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:38.328787 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:38.329450 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:38.329608 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:38.330945 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:38.330954 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:38.330969 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:38.331524 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:38.331604 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:38.331628 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:38.331766 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:38.332045 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 20:20:38.332370 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":405,"threads_started":1}
I20260923 20:20:38.335980 26376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf: No bootstrap required, opened a new log
I20260923 20:20:38.337313 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:38.345819 26376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d09d0942844146963fe7c18defffcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46461 } }
I20260923 20:20:38.346248 26376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:38.346391 26376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d09d0942844146963fe7c18defffcf, State: Initialized, Role: FOLLOWER
I20260923 20:20:38.347123 26376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d09d0942844146963fe7c18defffcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46461 } }
I20260923 20:20:38.347438 26376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:38.347602 26376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:38.347774 26376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:38.350585 26376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d09d0942844146963fe7c18defffcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46461 } }
I20260923 20:20:38.351219 26376 leader_election.cc:302] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17d09d0942844146963fe7c18defffcf; no voters:
I20260923 20:20:38.351950 26376 leader_election.cc:288] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:38.352090 26386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:38.353245 26386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [term 1 LEADER]: Becoming Leader. State: Replica: 17d09d0942844146963fe7c18defffcf, State: Running, Role: LEADER
I20260923 20:20:38.353600 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:38.342858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51983 (local address 127.0.0.1:46461)
0923 20:20:38.343069 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:38.343075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:38.346852 (+ 3777us) server_negotiation.cc:407] Connection header received
0923 20:20:38.347054 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:38.347060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:38.347156 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:38.347278 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:38.347885 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:38.349051 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:38.350349 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:38.351039 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:38.351170 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:38.352577 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:38.352587 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:38.352595 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:38.352627 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:38.352722 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:38.352755 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:38.352840 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:38.353141 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 20:20:38.353264 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 20:20:38.353940 26386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d09d0942844146963fe7c18defffcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46461 } }
I20260923 20:20:38.354532 26376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:38.357129 26388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17d09d0942844146963fe7c18defffcf. Latest consensus state: current_term: 1 leader_uuid: "17d09d0942844146963fe7c18defffcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d09d0942844146963fe7c18defffcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46461 } } }
I20260923 20:20:38.357527 26388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:38.359261 26394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:38.358266 26387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17d09d0942844146963fe7c18defffcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d09d0942844146963fe7c18defffcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46461 } } }
I20260923 20:20:38.359488 26387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:38.365020 26394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:38.374387 26394 catalog_manager.cc:1347] Generated new cluster ID: d4c059aada304c5bae15e6bbf25f03ed
I20260923 20:20:38.374508 26394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:38.386795 26394 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:38.388217 26394 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:38.395318 26394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17d09d0942844146963fe7c18defffcf: Generated new TSK 0
I20260923 20:20:38.396330 26394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:38.414593 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26309
W20260923 20:20:38.415793 26371 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46035
W20260923 20:20:38.416646 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26309: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:20:38.416805 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 26309
2026-09-23T20:20:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1977 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:38 dist-test-slave-vl24 krb5kdc[26407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:38 dist-test-slave-vl24 krb5kdc[26407](info): set up 2 sockets
Sep 23 20:20:38 dist-test-slave-vl24 krb5kdc[26407](info): commencing operation
krb5kdc: starting...
W20260923 20:20:39.542512 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:39 dist-test-slave-vl24 krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:20:39.827231 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:40.183133 26421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:40.183560 26421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:40.183727 26421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:40.191051 26421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:40.191215 26421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:40.191326 26421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:40.191438 26421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:40.191514 26421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:40.201822 26421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:40.203255 26421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:40.204668 26421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:40.216771 26426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:40.216814 26427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:40.216850 26429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:40.217741 26421 server_base.cc:1034] running on GCE node
I20260923 20:20:40.218583 26421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:40.220021 26421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:40.221282 26421 hybrid_clock.cc:648] HybridClock initialized: now 1790194840221233 us; error 56 us; skew 500 ppm
I20260923 20:20:40.221707 26421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:40 dist-test-slave-vl24 krb5kdc[26407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:40.229378 26421 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:20:40.230958 26421 webserver.cc:460] Webserver started at http://127.0.0.1:44997/ using document root <none> and password file <none>
I20260923 20:20:40.231652 26421 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:40.231789 26421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:40.232091 26421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:40.235210 26421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "0b4d76534a7e49278318da11b7df8044"
format_stamp: "Formatted at 2026-09-23 20:20:40 on dist-test-slave-vl24"
server_key: "03b97d08506d583ba3e3a5137b938396"
server_key_iv: "d147d555ef5de5f0ebe50b6f63e7e87a"
server_key_version: "encryptionkey@0"
I20260923 20:20:40.235962 26421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "0b4d76534a7e49278318da11b7df8044"
format_stamp: "Formatted at 2026-09-23 20:20:40 on dist-test-slave-vl24"
server_key: "03b97d08506d583ba3e3a5137b938396"
server_key_iv: "d147d555ef5de5f0ebe50b6f63e7e87a"
server_key_version: "encryptionkey@0"
I20260923 20:20:40.241546 26421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:20:40.245275 26436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:40.246730 26421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:20:40.246868 26421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "0b4d76534a7e49278318da11b7df8044"
format_stamp: "Formatted at 2026-09-23 20:20:40 on dist-test-slave-vl24"
server_key: "03b97d08506d583ba3e3a5137b938396"
server_key_iv: "d147d555ef5de5f0ebe50b6f63e7e87a"
server_key_version: "encryptionkey@0"
I20260923 20:20:40.247054 26421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:40.252933 26421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:40.257019 26421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:40.257269 26421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:40.276906 26421 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42813
I20260923 20:20:40.277015 26487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42813 every 8 connection(s)
I20260923 20:20:40.278364 26421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:40.282187 26488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:40.285813 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 26421
I20260923 20:20:40.286012 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:40.286478 5453 external_mini_cluster.cc:1428] Setting key 299357227a47721189c98f3951b9a9bc
I20260923 20:20:40.295992 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:40.288359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58315 (local address 127.0.0.1:42813)
0923 20:20:40.289259 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:40.289275 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:40.289329 (+ 54us) server_negotiation.cc:407] Connection header received
0923 20:20:40.291422 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:40.291455 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:40.291973 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:40.292444 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:40.293882 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:40.293895 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:40.293923 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:40.293944 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:40.294029 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:40.294064 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:40.294298 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:40.294439 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 20:20:40.294652 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":384,"threads_started":1}
I20260923 20:20:40.296350 26488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044: Bootstrap starting.
I20260923 20:20:40.299952 26488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:40.300496 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:40.301549 26488 log.cc:824] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:40.305583 26488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044: No bootstrap required, opened a new log
I20260923 20:20:40.308745 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:40.306097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54971 (local address 127.0.0.1:42813)
0923 20:20:40.306295 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:40.306301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:40.306670 (+ 369us) server_negotiation.cc:407] Connection header received
0923 20:20:40.306831 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:40.306836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:40.306898 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:40.307027 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:40.308022 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:40.308028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:40.308032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:40.308054 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:40.308109 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:40.308135 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:40.308268 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:40.308319 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 20:20:40.308387 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:20:40.314389 26488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4d76534a7e49278318da11b7df8044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42813 } }
I20260923 20:20:40.314782 26488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:40.314925 26488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4d76534a7e49278318da11b7df8044, State: Initialized, Role: FOLLOWER
I20260923 20:20:40.315579 26488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4d76534a7e49278318da11b7df8044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42813 } }
I20260923 20:20:40.315825 26488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:40.315979 26488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:40.316143 26488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:40.318600 26488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4d76534a7e49278318da11b7df8044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42813 } }
I20260923 20:20:40.319152 26488 leader_election.cc:302] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b4d76534a7e49278318da11b7df8044; no voters:
I20260923 20:20:40.319787 26488 leader_election.cc:288] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:40.319974 26498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:40.320789 26498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [term 1 LEADER]: Becoming Leader. State: Replica: 0b4d76534a7e49278318da11b7df8044, State: Running, Role: LEADER
I20260923 20:20:40.321403 26498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4d76534a7e49278318da11b7df8044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42813 } }
I20260923 20:20:40.322001 26488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:40.324592 26499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b4d76534a7e49278318da11b7df8044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4d76534a7e49278318da11b7df8044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42813 } } }
I20260923 20:20:40.325203 26499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:40.325556 26500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b4d76534a7e49278318da11b7df8044. Latest consensus state: current_term: 1 leader_uuid: "0b4d76534a7e49278318da11b7df8044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4d76534a7e49278318da11b7df8044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42813 } } }
I20260923 20:20:40.325860 26500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:40.331398 26504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:40.337369 26504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:40.346716 26504 catalog_manager.cc:1347] Generated new cluster ID: 97eff82fbec04353bdd855a2926a2044
I20260923 20:20:40.346854 26504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:40.382936 26504 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:40.383535 26504 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:40.395020 26504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b4d76534a7e49278318da11b7df8044: Generated new TSK 0
I20260923 20:20:40.396165 26504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:40.412289 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26421
W20260923 20:20:40.413467 26483 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58315
W20260923 20:20:40.414499 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26421: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:20:40.414696 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 26421
2026-09-23T20:20:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:40 dist-test-slave-vl24 krb5kdc[26519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:40 dist-test-slave-vl24 krb5kdc[26519](info): set up 2 sockets
Sep 23 20:20:40 dist-test-slave-vl24 krb5kdc[26519](info): commencing operation
krb5kdc: starting...
W20260923 20:20:41.546214 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:41 dist-test-slave-vl24 krb5kdc[26519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:20:41.837864 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:42.179878 26532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:42.180171 26532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:42.180262 26532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:42.187697 26532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:42.187820 26532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:42.187871 26532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:42.187937 26532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:42.187983 26532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:42.198144 26532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:42.199457 26532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:42.200816 26532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:42.212919 26537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:42.212980 26538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:42.213488 26540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:42.213997 26532 server_base.cc:1034] running on GCE node
I20260923 20:20:42.214787 26532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:42.216259 26532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:42.217547 26532 hybrid_clock.cc:648] HybridClock initialized: now 1790194842217486 us; error 68 us; skew 500 ppm
I20260923 20:20:42.217948 26532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:42 dist-test-slave-vl24 krb5kdc[26519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:42.221288 26532 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:20:42.222882 26532 webserver.cc:460] Webserver started at http://127.0.0.1:35893/ using document root <none> and password file <none>
I20260923 20:20:42.223604 26532 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:42.223745 26532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:42.224066 26532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:42.227146 26532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "e0cf1d2d4c6e40369bfe926fb9038448"
format_stamp: "Formatted at 2026-09-23 20:20:42 on dist-test-slave-vl24"
server_key: "b4f6d82274292d75577fe65f37a52d49"
server_key_iv: "2709a637bdb8bbfd588e1a5cabaad187"
server_key_version: "encryptionkey@0"
I20260923 20:20:42.227916 26532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "e0cf1d2d4c6e40369bfe926fb9038448"
format_stamp: "Formatted at 2026-09-23 20:20:42 on dist-test-slave-vl24"
server_key: "b4f6d82274292d75577fe65f37a52d49"
server_key_iv: "2709a637bdb8bbfd588e1a5cabaad187"
server_key_version: "encryptionkey@0"
I20260923 20:20:42.233610 26532 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 20:20:42.237493 26547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:42.238925 26532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 20:20:42.239131 26532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "e0cf1d2d4c6e40369bfe926fb9038448"
format_stamp: "Formatted at 2026-09-23 20:20:42 on dist-test-slave-vl24"
server_key: "b4f6d82274292d75577fe65f37a52d49"
server_key_iv: "2709a637bdb8bbfd588e1a5cabaad187"
server_key_version: "encryptionkey@0"
I20260923 20:20:42.239358 26532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:42.259440 26532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:42.263441 26532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:42.263757 26532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:42.282872 26532 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34009
I20260923 20:20:42.282891 26598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34009 every 8 connection(s)
I20260923 20:20:42.284343 26532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:42.285602 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 26532
I20260923 20:20:42.285789 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:42.286237 5453 external_mini_cluster.cc:1428] Setting key 9edcf2085e03075f7d55cc751d8f0763
I20260923 20:20:42.288847 26599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:20:42 dist-test-slave-vl24 krb5kdc[26519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:20:42.302088 26599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448: Bootstrap starting.
I20260923 20:20:42.306334 26599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:42.307634 26599 log.cc:824] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:42.311553 26599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448: No bootstrap required, opened a new log
I20260923 20:20:42.311684 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:42.288002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:36335 (local address 127.0.0.1:34009)
0923 20:20:42.288787 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:42.288802 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:42.288860 (+ 58us) server_negotiation.cc:407] Connection header received
0923 20:20:42.291205 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:42.291232 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:42.291733 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:42.292209 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:42.293691 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:42.295770 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:42.297170 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:42.298189 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:42.298373 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:42.302863 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:42.302874 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:42.302893 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:42.302958 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:42.306976 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:42.307847 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:42.307855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:42.307863 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:42.308081 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:42.308568 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:42.308576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:42.308582 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:42.309223 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:42.309579 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:42.309997 (+ 418us) server_negotiation.cc:299] Negotiation successful
0923 20:20:42.310341 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":329,"threads_started":1}
I20260923 20:20:42.315891 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:42.320438 26599 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0cf1d2d4c6e40369bfe926fb9038448" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260923 20:20:42.321015 26599 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:42.321146 26599 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cf1d2d4c6e40369bfe926fb9038448, State: Initialized, Role: FOLLOWER
I20260923 20:20:42.321784 26599 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0cf1d2d4c6e40369bfe926fb9038448" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260923 20:20:42.322026 26599 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:42.322171 26599 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:42.322355 26599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:42.324733 26599 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0cf1d2d4c6e40369bfe926fb9038448" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260923 20:20:42.325256 26599 leader_election.cc:302] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0cf1d2d4c6e40369bfe926fb9038448; no voters:
I20260923 20:20:42.325980 26599 leader_election.cc:288] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:42.326098 26609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:42.326817 26609 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [term 1 LEADER]: Becoming Leader. State: Replica: e0cf1d2d4c6e40369bfe926fb9038448, State: Running, Role: LEADER
I20260923 20:20:42.327488 26609 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0cf1d2d4c6e40369bfe926fb9038448" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260923 20:20:42.328682 26599 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:42.329700 26611 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0cf1d2d4c6e40369bfe926fb9038448. Latest consensus state: current_term: 1 leader_uuid: "e0cf1d2d4c6e40369bfe926fb9038448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0cf1d2d4c6e40369bfe926fb9038448" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } } }
I20260923 20:20:42.330180 26611 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:42.331060 26610 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0cf1d2d4c6e40369bfe926fb9038448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0cf1d2d4c6e40369bfe926fb9038448" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } } }
I20260923 20:20:42.331533 26610 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:42.339298 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:42.320306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:51935 (local address 127.0.0.1:34009)
0923 20:20:42.320471 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:42.320475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:42.320982 (+ 507us) server_negotiation.cc:407] Connection header received
0923 20:20:42.321189 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:42.321195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:42.321294 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:42.321421 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:42.323063 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:42.323896 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:42.325281 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:42.325799 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:42.325966 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:42.328818 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:42.328826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:42.328832 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:42.328878 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:42.333650 (+ 4772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:42.335113 (+ 1463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:42.335121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:42.335128 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:42.335320 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:42.336071 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:42.336079 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:42.336085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:42.336635 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:42.338213 (+ 1578us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:42.338452 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 20:20:42.338725 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:20:42.341928 26615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:42.347501 26615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:42.358548 26615 catalog_manager.cc:1347] Generated new cluster ID: da6549002b4b43a7815aa5f0b363d07c
I20260923 20:20:42.358734 26615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:42.385440 26615 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:42.386814 26615 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:42.394062 26615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0cf1d2d4c6e40369bfe926fb9038448: Generated new TSK 0
I20260923 20:20:42.394991 26615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:42.411854 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26532
I20260923 20:20:42.413635 26594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:42.512374 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 26532
W20260923 20:20:42.531106 26517 connection.cc:535] client connection to 127.0.0.1:34009 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34009): Connection reset by peer (error 104)
2026-09-23T20:20:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2096 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:42 dist-test-slave-vl24 krb5kdc[26632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:42 dist-test-slave-vl24 krb5kdc[26632](info): set up 2 sockets
Sep 23 20:20:42 dist-test-slave-vl24 krb5kdc[26632](info): commencing operation
krb5kdc: starting...
W20260923 20:20:43.626225 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:43 dist-test-slave-vl24 krb5kdc[26632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:20:43.917945 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:44.263072 26645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:44.263396 26645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:44.263583 26645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:44.271050 26645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:44.271214 26645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:44.271310 26645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:44.271417 26645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:44.271507 26645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:44.281958 26645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:44.283324 26645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:44.284739 26645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:44.298391 26650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:44.298477 26651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:44.298449 26653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:44.299574 26645 server_base.cc:1034] running on GCE node
I20260923 20:20:44.300483 26645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:44.302136 26645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:44.303383 26645 hybrid_clock.cc:648] HybridClock initialized: now 1790194844303336 us; error 43 us; skew 500 ppm
I20260923 20:20:44.303777 26645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:44 dist-test-slave-vl24 krb5kdc[26632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:44.307322 26645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:20:44.309091 26645 webserver.cc:460] Webserver started at http://127.0.0.1:34633/ using document root <none> and password file <none>
I20260923 20:20:44.309839 26645 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:44.309991 26645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:44.310307 26645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:44.313491 26645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "8b655a1371e34f3d8761dc102ff12d10"
format_stamp: "Formatted at 2026-09-23 20:20:44 on dist-test-slave-vl24"
server_key: "7db3d3830a06419d193717edc34f317f"
server_key_iv: "68f07ccf8e3193f9fdb6acafc3c32ef9"
server_key_version: "encryptionkey@0"
I20260923 20:20:44.314293 26645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "8b655a1371e34f3d8761dc102ff12d10"
format_stamp: "Formatted at 2026-09-23 20:20:44 on dist-test-slave-vl24"
server_key: "7db3d3830a06419d193717edc34f317f"
server_key_iv: "68f07ccf8e3193f9fdb6acafc3c32ef9"
server_key_version: "encryptionkey@0"
I20260923 20:20:44.319840 26645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:20:44.323843 26660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:44.325453 26645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 20:20:44.325685 26645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "8b655a1371e34f3d8761dc102ff12d10"
format_stamp: "Formatted at 2026-09-23 20:20:44 on dist-test-slave-vl24"
server_key: "7db3d3830a06419d193717edc34f317f"
server_key_iv: "68f07ccf8e3193f9fdb6acafc3c32ef9"
server_key_version: "encryptionkey@0"
I20260923 20:20:44.325917 26645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:44.347733 26645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:44.352326 26645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:44.352661 26645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:44.372043 26645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34243
I20260923 20:20:44.372083 26711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34243 every 8 connection(s)
I20260923 20:20:44.373481 26645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:44.377310 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 26645
I20260923 20:20:44.377504 26712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:44.377519 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:44.378060 5453 external_mini_cluster.cc:1428] Setting key 5799f9a9202c6bb7331d3dc7e9651b55
I20260923 20:20:44.390925 26712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10: Bootstrap starting.
I20260923 20:20:44.391443 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:44.379963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:43739 (local address 127.0.0.1:34243)
0923 20:20:44.380827 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:44.380838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:44.380877 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:20:44.382477 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:44.382499 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:44.382885 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:44.383287 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:44.383855 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:44.385303 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:44.386696 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:44.387398 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:44.387626 (+ 228us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:44.389080 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:44.389087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:44.389103 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:44.389668 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:44.389744 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:44.389768 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:44.389936 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:44.390224 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 20:20:44.390403 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":353,"threads_started":1}
I20260923 20:20:44.394405 26712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:44.395339 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:44.395625 26712 log.cc:824] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:44.398693 26712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10: No bootstrap required, opened a new log
I20260923 20:20:44.406737 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:44.399892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:43717 (local address 127.0.0.1:34243)
0923 20:20:44.400055 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:44.400059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:44.401068 (+ 1009us) server_negotiation.cc:407] Connection header received
0923 20:20:44.401294 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:44.401298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:44.401361 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:44.401483 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:44.402002 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:44.402776 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:44.404121 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:44.404657 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:44.404815 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:44.405997 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:44.406001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:44.406005 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:44.406026 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:44.406062 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:44.406080 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:44.406203 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:44.406399 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 20:20:44.406452 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:20:44.407778 26712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b655a1371e34f3d8761dc102ff12d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } }
I20260923 20:20:44.408243 26712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:44.408385 26712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b655a1371e34f3d8761dc102ff12d10, State: Initialized, Role: FOLLOWER
I20260923 20:20:44.408977 26712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b655a1371e34f3d8761dc102ff12d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } }
I20260923 20:20:44.409185 26712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:44.409283 26712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:44.409411 26712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:44.412045 26712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b655a1371e34f3d8761dc102ff12d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } }
I20260923 20:20:44.412611 26712 leader_election.cc:302] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b655a1371e34f3d8761dc102ff12d10; no voters:
I20260923 20:20:44.413143 26712 leader_election.cc:288] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:44.413311 26722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:44.414116 26722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [term 1 LEADER]: Becoming Leader. State: Replica: 8b655a1371e34f3d8761dc102ff12d10, State: Running, Role: LEADER
I20260923 20:20:44.414635 26722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b655a1371e34f3d8761dc102ff12d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } }
I20260923 20:20:44.415361 26712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:44.417467 26723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b655a1371e34f3d8761dc102ff12d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b655a1371e34f3d8761dc102ff12d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } } }
I20260923 20:20:44.417997 26723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:44.417492 26724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b655a1371e34f3d8761dc102ff12d10. Latest consensus state: current_term: 1 leader_uuid: "8b655a1371e34f3d8761dc102ff12d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b655a1371e34f3d8761dc102ff12d10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } } }
I20260923 20:20:44.419467 26724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:44.420924 26727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:44.427625 26727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:44.437211 26727 catalog_manager.cc:1347] Generated new cluster ID: 87110818860a4476a54705f9253f91c3
I20260923 20:20:44.437345 26727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:44.460897 26727 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:44.462338 26727 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:44.474073 26727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b655a1371e34f3d8761dc102ff12d10: Generated new TSK 0
I20260923 20:20:44.475082 26727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:44.495685 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26645
W20260923 20:20:44.497115 26707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.30:43739
W20260923 20:20:44.497895 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26645: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:20:44.498062 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 26645
W20260923 20:20:44.513809 26630 connection.cc:535] client connection to 127.0.0.1:34243 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34243): Connection reset by peer (error 104)
2026-09-23T20:20:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1982 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:44 dist-test-slave-vl24 krb5kdc[26743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:44 dist-test-slave-vl24 krb5kdc[26743](info): set up 2 sockets
Sep 23 20:20:44 dist-test-slave-vl24 krb5kdc[26743](info): commencing operation
krb5kdc: starting...
W20260923 20:20:45.650271 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:45 dist-test-slave-vl24 krb5kdc[26743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:20:45.942262 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:46.288143 26756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:46.288518 26756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:46.288744 26756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:46.296264 26756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:46.296434 26756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:46.296597 26756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:46.296703 26756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:46.296798 26756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:46.307087 26756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:46.308496 26756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:46.309885 26756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:46.322528 26762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:46.322566 26765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:46.322772 26761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:46.323570 26756 server_base.cc:1034] running on GCE node
I20260923 20:20:46.324493 26756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:46.326205 26756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:46.327455 26756 hybrid_clock.cc:648] HybridClock initialized: now 1790194846327410 us; error 45 us; skew 500 ppm
I20260923 20:20:46.327838 26756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:46 dist-test-slave-vl24 krb5kdc[26743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:46.331305 26756 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:20:46.333092 26756 webserver.cc:460] Webserver started at http://127.0.0.1:40895/ using document root <none> and password file <none>
I20260923 20:20:46.333804 26756 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:46.333951 26756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:46.334257 26756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:46.337533 26756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "26f91fc14c8e426eb168d3ee149edbd6"
format_stamp: "Formatted at 2026-09-23 20:20:46 on dist-test-slave-vl24"
server_key: "93b0dfa7256bf1d57a92accf6dfb70fd"
server_key_iv: "2e5725e89e94b7ebd8b1da32605c4bd0"
server_key_version: "encryptionkey@0"
I20260923 20:20:46.338326 26756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "26f91fc14c8e426eb168d3ee149edbd6"
format_stamp: "Formatted at 2026-09-23 20:20:46 on dist-test-slave-vl24"
server_key: "93b0dfa7256bf1d57a92accf6dfb70fd"
server_key_iv: "2e5725e89e94b7ebd8b1da32605c4bd0"
server_key_version: "encryptionkey@0"
I20260923 20:20:46.343657 26756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:20:46.347172 26772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:46.348505 26756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 20:20:46.348764 26756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "26f91fc14c8e426eb168d3ee149edbd6"
format_stamp: "Formatted at 2026-09-23 20:20:46 on dist-test-slave-vl24"
server_key: "93b0dfa7256bf1d57a92accf6dfb70fd"
server_key_iv: "2e5725e89e94b7ebd8b1da32605c4bd0"
server_key_version: "encryptionkey@0"
I20260923 20:20:46.348979 26756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:46.354915 26756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:46.359445 26756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:46.359757 26756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:46.378885 26756 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43341
I20260923 20:20:46.378945 26823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43341 every 8 connection(s)
I20260923 20:20:46.380384 26756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:46.380859 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 26756
I20260923 20:20:46.381052 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:46.381685 5453 external_mini_cluster.cc:1428] Setting key b99af58d0f41dbff50b886e547d15ad7
I20260923 20:20:46.385308 26824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:46.390596 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:46.383507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:41125 (local address 127.0.0.1:43341)
0923 20:20:46.384281 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:46.384296 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:46.384353 (+ 57us) server_negotiation.cc:407] Connection header received
0923 20:20:46.386741 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:46.386768 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:46.387190 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:46.387625 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:46.388648 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:46.388661 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:46.388677 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:46.388694 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:46.388777 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:46.388802 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:46.389024 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:46.389163 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 20:20:46.389359 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":324,"threads_started":1}
I20260923 20:20:46.394459 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:46.398465 26824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6: Bootstrap starting.
I20260923 20:20:46.401968 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:46.399127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:60005 (local address 127.0.0.1:43341)
0923 20:20:46.399322 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:46.399330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:46.399883 (+ 553us) server_negotiation.cc:407] Connection header received
0923 20:20:46.400058 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:46.400062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:46.400118 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:46.400208 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:46.401264 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:46.401268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:46.401271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:46.401287 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:46.401329 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:46.401344 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:46.401438 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:46.401575 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 20:20:46.401625 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 20:20:46.402676 26824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:46.403936 26824 log.cc:824] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:46.406787 26824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6: No bootstrap required, opened a new log
I20260923 20:20:46.415370 26824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f91fc14c8e426eb168d3ee149edbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } }
I20260923 20:20:46.415792 26824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:46.415910 26824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f91fc14c8e426eb168d3ee149edbd6, State: Initialized, Role: FOLLOWER
I20260923 20:20:46.416666 26824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f91fc14c8e426eb168d3ee149edbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } }
I20260923 20:20:46.416996 26824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:46.417142 26824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:46.417338 26824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:46.420464 26824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f91fc14c8e426eb168d3ee149edbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } }
I20260923 20:20:46.421176 26824 leader_election.cc:302] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26f91fc14c8e426eb168d3ee149edbd6; no voters:
I20260923 20:20:46.421943 26824 leader_election.cc:288] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:46.422132 26834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:46.424126 26824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:46.424167 26834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [term 1 LEADER]: Becoming Leader. State: Replica: 26f91fc14c8e426eb168d3ee149edbd6, State: Running, Role: LEADER
I20260923 20:20:46.424979 26834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f91fc14c8e426eb168d3ee149edbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } }
I20260923 20:20:46.430274 26835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26f91fc14c8e426eb168d3ee149edbd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f91fc14c8e426eb168d3ee149edbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } } }
I20260923 20:20:46.430615 26835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:46.431039 26836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26f91fc14c8e426eb168d3ee149edbd6. Latest consensus state: current_term: 1 leader_uuid: "26f91fc14c8e426eb168d3ee149edbd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26f91fc14c8e426eb168d3ee149edbd6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } } }
I20260923 20:20:46.431314 26836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6 [sys.catalog]: This master's current role is: LEADER
W20260923 20:20:46.440798 26850 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:20:46.440910 26850 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:20:46.441045 26847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:46.443281 26847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:46.452721 26847 catalog_manager.cc:1347] Generated new cluster ID: d41eadedb55643c8b7e575a5c46e422a
I20260923 20:20:46.452859 26847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:46.470252 26847 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:46.470731 26847 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:46.481137 26847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26f91fc14c8e426eb168d3ee149edbd6: Generated new TSK 0
I20260923 20:20:46.482227 26847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:46.493044 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26756
W20260923 20:20:46.494681 26819 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.30:41125
W20260923 20:20:46.495496 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26756: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:20:46.495658 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 26756
2026-09-23T20:20:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:46 dist-test-slave-vl24 krb5kdc[26855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:46 dist-test-slave-vl24 krb5kdc[26855](info): set up 2 sockets
Sep 23 20:20:46 dist-test-slave-vl24 krb5kdc[26855](info): commencing operation
krb5kdc: starting...
W20260923 20:20:47.618445 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.005s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:47 dist-test-slave-vl24 krb5kdc[26855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:20:47.908277 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:48.251963 26868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:48.252276 26868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:48.252368 26868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:48.260116 26868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:48.260244 26868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:48.260344 26868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:48.260447 26868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:48.260533 26868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:48.270975 26868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:48.272334 26868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:48.273717 26868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:48.286031 26873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:48.286012 26874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:48.285969 26876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:48.286844 26868 server_base.cc:1034] running on GCE node
I20260923 20:20:48.287691 26868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:48.289315 26868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:48.290553 26868 hybrid_clock.cc:648] HybridClock initialized: now 1790194848290500 us; error 51 us; skew 500 ppm
I20260923 20:20:48.290937 26868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:48 dist-test-slave-vl24 krb5kdc[26855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:48.294269 26868 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:20:48.295821 26868 webserver.cc:460] Webserver started at http://127.0.0.1:34563/ using document root <none> and password file <none>
I20260923 20:20:48.296623 26868 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:48.296769 26868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:48.297087 26868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:48.300123 26868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "eaae5a87c81d4a20ac993b5ebcf5df61"
format_stamp: "Formatted at 2026-09-23 20:20:48 on dist-test-slave-vl24"
server_key: "f82ab5b2ae072a3c3a413f869c26aa97"
server_key_iv: "419f7053a929c3da15788e8bddcdc0c4"
server_key_version: "encryptionkey@0"
I20260923 20:20:48.300972 26868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "eaae5a87c81d4a20ac993b5ebcf5df61"
format_stamp: "Formatted at 2026-09-23 20:20:48 on dist-test-slave-vl24"
server_key: "f82ab5b2ae072a3c3a413f869c26aa97"
server_key_iv: "419f7053a929c3da15788e8bddcdc0c4"
server_key_version: "encryptionkey@0"
I20260923 20:20:48.306208 26868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 20:20:48.309659 26883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:48.310971 26868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 20:20:48.311162 26868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "eaae5a87c81d4a20ac993b5ebcf5df61"
format_stamp: "Formatted at 2026-09-23 20:20:48 on dist-test-slave-vl24"
server_key: "f82ab5b2ae072a3c3a413f869c26aa97"
server_key_iv: "419f7053a929c3da15788e8bddcdc0c4"
server_key_version: "encryptionkey@0"
I20260923 20:20:48.311364 26868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:48.340806 26868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:48.345407 26868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:48.345705 26868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:48.365110 26868 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33641
I20260923 20:20:48.365144 26934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33641 every 8 connection(s)
I20260923 20:20:48.366545 26868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:48.367071 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 26868
I20260923 20:20:48.367281 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:48.367887 5453 external_mini_cluster.cc:1428] Setting key d2009f98842d0016106b15acb60c80bd
I20260923 20:20:48.370972 26935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 20:20:48 dist-test-slave-vl24 krb5kdc[26855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 20:20:48.384289 26935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61: Bootstrap starting.
I20260923 20:20:48.387974 26935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:48.389458 26935 log.cc:824] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:48.392465 26938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:48.369857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:32923 (local address 127.0.0.1:33641)
0923 20:20:48.370665 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:48.370677 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:48.370716 (+ 39us) server_negotiation.cc:407] Connection header received
0923 20:20:48.372386 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:48.372410 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:48.372849 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:48.373183 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:48.374983 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:48.376314 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:48.378135 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:48.378781 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:48.378937 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:48.383812 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:48.383819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:48.383834 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:48.383873 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:48.387359 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:48.388355 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:48.388368 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:48.388378 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:48.388666 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:48.389155 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:48.389160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:48.389163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:48.389893 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:48.390291 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:48.390747 (+ 456us) server_negotiation.cc:299] Negotiation successful
0923 20:20:48.391063 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":347,"threads_started":1}
I20260923 20:20:48.393394 26935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61: No bootstrap required, opened a new log
I20260923 20:20:48.396977 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:48.403155 26935 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaae5a87c81d4a20ac993b5ebcf5df61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } }
I20260923 20:20:48.403585 26935 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:48.403697 26935 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaae5a87c81d4a20ac993b5ebcf5df61, State: Initialized, Role: FOLLOWER
I20260923 20:20:48.404340 26935 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaae5a87c81d4a20ac993b5ebcf5df61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } }
I20260923 20:20:48.404624 26935 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:48.404819 26935 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:48.405030 26935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:48.407306 26935 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaae5a87c81d4a20ac993b5ebcf5df61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } }
I20260923 20:20:48.407886 26935 leader_election.cc:302] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaae5a87c81d4a20ac993b5ebcf5df61; no voters:
I20260923 20:20:48.408594 26935 leader_election.cc:288] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:48.408723 26946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:48.411407 26935 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:48.411434 26946 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [term 1 LEADER]: Becoming Leader. State: Replica: eaae5a87c81d4a20ac993b5ebcf5df61, State: Running, Role: LEADER
I20260923 20:20:48.412088 26946 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaae5a87c81d4a20ac993b5ebcf5df61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } }
I20260923 20:20:48.417484 26948 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaae5a87c81d4a20ac993b5ebcf5df61. Latest consensus state: current_term: 1 leader_uuid: "eaae5a87c81d4a20ac993b5ebcf5df61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaae5a87c81d4a20ac993b5ebcf5df61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } } }
I20260923 20:20:48.418056 26948 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:48.419515 26947 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaae5a87c81d4a20ac993b5ebcf5df61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaae5a87c81d4a20ac993b5ebcf5df61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33641 } } }
I20260923 20:20:48.420046 26947 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:48.423161 26938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:48.401645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:52211 (local address 127.0.0.1:33641)
0923 20:20:48.401964 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:48.401972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:48.402357 (+ 385us) server_negotiation.cc:407] Connection header received
0923 20:20:48.402594 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:48.402599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:48.402671 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:48.402793 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:48.404758 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:48.405777 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:48.407183 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:48.407730 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:48.407885 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:48.411194 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:48.411205 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:48.411213 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:48.411279 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:48.415952 (+ 4673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:48.418640 (+ 2688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:48.418650 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:48.418656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:48.419108 (+ 452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:48.419887 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:48.419896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:48.419902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:48.420438 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:48.422335 (+ 1897us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:48.422547 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 20:20:48.422771 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260923 20:20:48.425829 26955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:48.431634 26955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:48.441370 26955 catalog_manager.cc:1347] Generated new cluster ID: 12166f0e1fb54ea78e6ea8623ebafc84
I20260923 20:20:48.441505 26955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:48.463263 26955 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:48.464820 26955 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:48.472543 26955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaae5a87c81d4a20ac993b5ebcf5df61: Generated new TSK 0
I20260923 20:20:48.473464 26955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:48.488870 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26868
I20260923 20:20:48.490039 26930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:48.582831 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 26868
W20260923 20:20:48.601069 26853 connection.cc:535] client connection to 127.0.0.1:33641 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33641): Connection reset by peer (error 104)
2026-09-23T20:20:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2086 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:48 dist-test-slave-vl24 krb5kdc[26968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:48 dist-test-slave-vl24 krb5kdc[26968](info): set up 2 sockets
Sep 23 20:20:48 dist-test-slave-vl24 krb5kdc[26968](info): commencing operation
krb5kdc: starting...
W20260923 20:20:49.694243 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:49 dist-test-slave-vl24 krb5kdc[26968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:20:49.983882 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:50.328109 26981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:50.328418 26981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:50.328629 26981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:50.335988 26981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:50.336108 26981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:50.336197 26981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:50.336299 26981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:50.336373 26981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:50.346612 26981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:50.347990 26981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:50.349346 26981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:50.361620 26986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:50.361595 26989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:50.361701 26987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:50.362614 26981 server_base.cc:1034] running on GCE node
I20260923 20:20:50.363436 26981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:50.364904 26981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:50.366156 26981 hybrid_clock.cc:648] HybridClock initialized: now 1790194850366116 us; error 47 us; skew 500 ppm
I20260923 20:20:50.366542 26981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:50 dist-test-slave-vl24 krb5kdc[26968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:50.369700 26981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:20:50.371268 26981 webserver.cc:460] Webserver started at http://127.0.0.1:40631/ using document root <none> and password file <none>
I20260923 20:20:50.371981 26981 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:50.372128 26981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:50.372437 26981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:50.375586 26981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "43ae8b9e44514d98a0d510d8cde6fcf8"
format_stamp: "Formatted at 2026-09-23 20:20:50 on dist-test-slave-vl24"
server_key: "22b9c208edd9c04856ad6b61d535927e"
server_key_iv: "6014d08708a36d799b0f6a93eb617787"
server_key_version: "encryptionkey@0"
I20260923 20:20:50.376375 26981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "43ae8b9e44514d98a0d510d8cde6fcf8"
format_stamp: "Formatted at 2026-09-23 20:20:50 on dist-test-slave-vl24"
server_key: "22b9c208edd9c04856ad6b61d535927e"
server_key_iv: "6014d08708a36d799b0f6a93eb617787"
server_key_version: "encryptionkey@0"
I20260923 20:20:50.381778 26981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:20:50.385442 26996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:50.386792 26981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:20:50.386988 26981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "43ae8b9e44514d98a0d510d8cde6fcf8"
format_stamp: "Formatted at 2026-09-23 20:20:50 on dist-test-slave-vl24"
server_key: "22b9c208edd9c04856ad6b61d535927e"
server_key_iv: "6014d08708a36d799b0f6a93eb617787"
server_key_version: "encryptionkey@0"
I20260923 20:20:50.387193 26981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:50.422847 26981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:50.427035 26981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:50.427325 26981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:50.446903 26981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38239
I20260923 20:20:50.446951 27047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38239 every 8 connection(s)
I20260923 20:20:50.448516 26981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:50.452389 27048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:50.452936 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 26981
I20260923 20:20:50.453141 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:50.453608 5453 external_mini_cluster.cc:1428] Setting key 0893e822c7f3ea627c87414bff1fb854
I20260923 20:20:50.466483 27048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8: Bootstrap starting.
I20260923 20:20:50.468174 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:50.455448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:51697 (local address 127.0.0.1:38239)
0923 20:20:50.456359 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:50.456372 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:50.456418 (+ 46us) server_negotiation.cc:407] Connection header received
0923 20:20:50.458649 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:50.458674 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:50.459090 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:50.459434 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:50.460057 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:50.461397 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:50.463354 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:50.464099 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:50.464239 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:50.465636 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:50.465643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:50.465659 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:50.466205 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:50.466290 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:50.466315 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:50.466431 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:50.466881 (+ 450us) server_negotiation.cc:299] Negotiation successful
0923 20:20:50.467103 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":460,"threads_started":1}
I20260923 20:20:50.469913 27048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:50.471171 27048 log.cc:824] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:50.472514 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:50.474159 27048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8: No bootstrap required, opened a new log
I20260923 20:20:50.484277 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:50.477098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:44797 (local address 127.0.0.1:38239)
0923 20:20:50.477310 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:50.477315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:50.477623 (+ 308us) server_negotiation.cc:407] Connection header received
0923 20:20:50.477833 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:50.477837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:50.477904 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:50.477988 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:50.478580 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:50.479415 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:50.481363 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:50.481967 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:50.482081 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:50.483426 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:50.483431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:50.483434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:50.483452 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:50.483505 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:50.483521 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:50.483606 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:50.483835 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 20:20:50.483906 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 20:20:50.484053 27048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ae8b9e44514d98a0d510d8cde6fcf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38239 } }
I20260923 20:20:50.484494 27048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:50.484648 27048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ae8b9e44514d98a0d510d8cde6fcf8, State: Initialized, Role: FOLLOWER
I20260923 20:20:50.485301 27048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ae8b9e44514d98a0d510d8cde6fcf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38239 } }
I20260923 20:20:50.485569 27048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:50.485744 27048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:50.485921 27048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:50.488153 27048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ae8b9e44514d98a0d510d8cde6fcf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38239 } }
I20260923 20:20:50.488811 27048 leader_election.cc:302] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43ae8b9e44514d98a0d510d8cde6fcf8; no voters:
I20260923 20:20:50.489544 27048 leader_election.cc:288] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:50.489722 27058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:50.490518 27058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [term 1 LEADER]: Becoming Leader. State: Replica: 43ae8b9e44514d98a0d510d8cde6fcf8, State: Running, Role: LEADER
I20260923 20:20:50.491137 27058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ae8b9e44514d98a0d510d8cde6fcf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38239 } }
I20260923 20:20:50.491870 27048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:50.494181 27060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43ae8b9e44514d98a0d510d8cde6fcf8. Latest consensus state: current_term: 1 leader_uuid: "43ae8b9e44514d98a0d510d8cde6fcf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ae8b9e44514d98a0d510d8cde6fcf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38239 } } }
I20260923 20:20:50.494575 27060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:50.495123 27059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43ae8b9e44514d98a0d510d8cde6fcf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ae8b9e44514d98a0d510d8cde6fcf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38239 } } }
I20260923 20:20:50.495491 27059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:50.496742 27065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:50.502760 27065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:50.513582 27065 catalog_manager.cc:1347] Generated new cluster ID: ec7e27e1a9fc4ed990cb1ead37ad58d9
I20260923 20:20:50.513754 27065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:50.526737 27065 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:50.528801 27065 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:50.542553 27065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43ae8b9e44514d98a0d510d8cde6fcf8: Generated new TSK 0
I20260923 20:20:50.543660 27065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:50.556702 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26981
W20260923 20:20:50.558351 27043 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.30:51697
W20260923 20:20:50.559300 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 26981: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:20:50.559511 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 26981
W20260923 20:20:50.579174 26966 connection.cc:535] client connection to 127.0.0.1:38239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38239): Connection reset by peer (error 104)
2026-09-23T20:20:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:50 dist-test-slave-vl24 krb5kdc[27079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:50 dist-test-slave-vl24 krb5kdc[27079](info): set up 2 sockets
Sep 23 20:20:50 dist-test-slave-vl24 krb5kdc[27079](info): commencing operation
krb5kdc: starting...
W20260923 20:20:51.722271 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:51 dist-test-slave-vl24 krb5kdc[27079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 20:20:52.010090 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:52.353648 27092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:52.353924 27092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:52.354015 27092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:52.361286 27092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:52.361395 27092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:52.361449 27092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:52.361505 27092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:52.361550 27092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:52.371436 27092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43817
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:52.372727 27092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:52.373981 27092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:52.386569 27097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:52.386613 27100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:52.386638 27098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:52.387421 27092 server_base.cc:1034] running on GCE node
I20260923 20:20:52.388279 27092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:52.389817 27092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:52.391047 27092 hybrid_clock.cc:648] HybridClock initialized: now 1790194852391007 us; error 36 us; skew 500 ppm
I20260923 20:20:52.391404 27092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:52 dist-test-slave-vl24 krb5kdc[27079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:52.394559 27092 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 20:20:52.396085 27092 webserver.cc:460] Webserver started at http://127.0.0.1:36445/ using document root <none> and password file <none>
I20260923 20:20:52.396862 27092 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:52.397006 27092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:52.397311 27092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:52.400273 27092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "365b53c5c22a4a6d9d31cc2cb0461318"
format_stamp: "Formatted at 2026-09-23 20:20:52 on dist-test-slave-vl24"
server_key: "37b09760e16d7a0ceae82443ae465fda"
server_key_iv: "555b1227d7a53496711cd615a704f90b"
server_key_version: "encryptionkey@0"
I20260923 20:20:52.401093 27092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "365b53c5c22a4a6d9d31cc2cb0461318"
format_stamp: "Formatted at 2026-09-23 20:20:52 on dist-test-slave-vl24"
server_key: "37b09760e16d7a0ceae82443ae465fda"
server_key_iv: "555b1227d7a53496711cd615a704f90b"
server_key_version: "encryptionkey@0"
I20260923 20:20:52.406194 27092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 20:20:52.409533 27107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:52.410907 27092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:20:52.411204 27092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "365b53c5c22a4a6d9d31cc2cb0461318"
format_stamp: "Formatted at 2026-09-23 20:20:52 on dist-test-slave-vl24"
server_key: "37b09760e16d7a0ceae82443ae465fda"
server_key_iv: "555b1227d7a53496711cd615a704f90b"
server_key_version: "encryptionkey@0"
I20260923 20:20:52.411433 27092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:52.417404 27092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:52.421543 27092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:52.421834 27092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:52.440745 27092 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43817
I20260923 20:20:52.440830 27159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43817 every 8 connection(s)
I20260923 20:20:52.442219 27092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:52.445943 27160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:52.448499 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 27092
I20260923 20:20:52.448685 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:52.449222 5453 external_mini_cluster.cc:1428] Setting key 1d9abd4acb475026c0c20e69846c75f0
I20260923 20:20:52.457104 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:52.451105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:50107 (local address 127.0.0.1:43817)
0923 20:20:52.451844 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:52.451855 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:52.451895 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:20:52.453665 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:52.453686 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:52.454065 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:52.454440 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:52.455463 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:52.455472 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:52.455484 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:52.455496 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:52.455555 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:52.455577 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:52.455809 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:52.455963 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 20:20:52.456128 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":325,"threads_started":1}
I20260923 20:20:52.458318 27160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318: Bootstrap starting.
I20260923 20:20:52.460637 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:52.461565 27160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:52.462574 27160 log.cc:824] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:52.465699 27160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318: No bootstrap required, opened a new log
I20260923 20:20:52.467440 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:52.464899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:44491 (local address 127.0.0.1:43817)
0923 20:20:52.465073 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:52.465080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:52.465539 (+ 459us) server_negotiation.cc:407] Connection header received
0923 20:20:52.465674 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:52.465680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:52.465752 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:52.465872 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:52.466757 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:52.466763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:52.466776 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:52.466794 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:52.466840 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:52.466860 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:52.466977 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:52.467100 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 20:20:52.467158 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 20:20:52.473778 27160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365b53c5c22a4a6d9d31cc2cb0461318" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43817 } }
I20260923 20:20:52.474164 27160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:52.474295 27160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365b53c5c22a4a6d9d31cc2cb0461318, State: Initialized, Role: FOLLOWER
I20260923 20:20:52.474918 27160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365b53c5c22a4a6d9d31cc2cb0461318" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43817 } }
I20260923 20:20:52.475275 27160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:52.475437 27160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:52.475605 27160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:52.478235 27160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365b53c5c22a4a6d9d31cc2cb0461318" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43817 } }
I20260923 20:20:52.478888 27160 leader_election.cc:302] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 365b53c5c22a4a6d9d31cc2cb0461318; no voters:
I20260923 20:20:52.479575 27160 leader_election.cc:288] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:52.479806 27170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:52.480597 27170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [term 1 LEADER]: Becoming Leader. State: Replica: 365b53c5c22a4a6d9d31cc2cb0461318, State: Running, Role: LEADER
I20260923 20:20:52.481328 27170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365b53c5c22a4a6d9d31cc2cb0461318" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43817 } }
I20260923 20:20:52.482120 27160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:52.484825 27172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 365b53c5c22a4a6d9d31cc2cb0461318. Latest consensus state: current_term: 1 leader_uuid: "365b53c5c22a4a6d9d31cc2cb0461318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365b53c5c22a4a6d9d31cc2cb0461318" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43817 } } }
I20260923 20:20:52.485652 27172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:52.485266 27171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "365b53c5c22a4a6d9d31cc2cb0461318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365b53c5c22a4a6d9d31cc2cb0461318" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43817 } } }
I20260923 20:20:52.486269 27171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:52.488040 27178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:52.492871 27178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:52.501549 27178 catalog_manager.cc:1347] Generated new cluster ID: 68ebbf4310de4a8c9cae656f75d5cf44
I20260923 20:20:52.501669 27178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:52.527770 27178 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:52.528216 27178 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:52.540987 27178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 365b53c5c22a4a6d9d31cc2cb0461318: Generated new TSK 0
I20260923 20:20:52.542148 27178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:52.561717 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27092
W20260923 20:20:52.562949 27155 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.30:50107
W20260923 20:20:52.563712 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27092: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:20:52.563874 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 27092
2026-09-23T20:20:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:52 dist-test-slave-vl24 krb5kdc[27191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:52 dist-test-slave-vl24 krb5kdc[27191](info): set up 2 sockets
Sep 23 20:20:52 dist-test-slave-vl24 krb5kdc[27191](info): commencing operation
krb5kdc: starting...
W20260923 20:20:53.686154 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:53 dist-test-slave-vl24 krb5kdc[27191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:20:53.973790 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:40065
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:40065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:54.317461 27204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:54.317835 27204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:54.318011 27204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:54.325456 27204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:54.325637 27204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:54.325747 27204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:54.325855 27204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:54.325947 27204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:54.336248 27204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:40065
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:40065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:54.337713 27204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:54.339031 27204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:54.350940 27209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:54.350924 27212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:54.350991 27210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:54.357291 27204 server_base.cc:1034] running on GCE node
I20260923 20:20:54.358161 27204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:54.359773 27204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:54.361008 27204 hybrid_clock.cc:648] HybridClock initialized: now 1790194854360961 us; error 43 us; skew 500 ppm
I20260923 20:20:54.361370 27204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:54 dist-test-slave-vl24 krb5kdc[27191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194854, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:54.364615 27204 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:20:54.366192 27204 webserver.cc:460] Webserver started at http://127.5.83.126:33283/ using document root <none> and password file <none>
I20260923 20:20:54.366875 27204 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:54.366989 27204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:54.367238 27204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:54.370251 27204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "82d9816d02184809a0510640c7562eef"
format_stamp: "Formatted at 2026-09-23 20:20:54 on dist-test-slave-vl24"
server_key: "10310e663cae489924c681194e91f6e3"
server_key_iv: "55ffa3235399d938c90bb1cd6ea282e6"
server_key_version: "encryptionkey@0"
I20260923 20:20:54.371012 27204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "82d9816d02184809a0510640c7562eef"
format_stamp: "Formatted at 2026-09-23 20:20:54 on dist-test-slave-vl24"
server_key: "10310e663cae489924c681194e91f6e3"
server_key_iv: "55ffa3235399d938c90bb1cd6ea282e6"
server_key_version: "encryptionkey@0"
I20260923 20:20:54.376129 27204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 20:20:54.379557 27219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:54.380863 27204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:20:54.381057 27204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "82d9816d02184809a0510640c7562eef"
format_stamp: "Formatted at 2026-09-23 20:20:54 on dist-test-slave-vl24"
server_key: "10310e663cae489924c681194e91f6e3"
server_key_iv: "55ffa3235399d938c90bb1cd6ea282e6"
server_key_version: "encryptionkey@0"
I20260923 20:20:54.381253 27204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:54.403515 27204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:54.408098 27204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:54.408396 27204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:54.427289 27204 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:40065
I20260923 20:20:54.427322 27271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:40065 every 8 connection(s)
I20260923 20:20:54.428735 27204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:54.431870 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 27204
I20260923 20:20:54.432044 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:54.432503 27272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:54.432507 5453 external_mini_cluster.cc:1428] Setting key 3a1b244c168462b30eecab3364bbdcc9
I20260923 20:20:54.446954 27272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef: Bootstrap starting.
Sep 23 20:20:54 dist-test-slave-vl24 krb5kdc[27191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:20:54.450599 27272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:54.451686 27272 log.cc:824] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:54.454514 27272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef: No bootstrap required, opened a new log
I20260923 20:20:54.456120 27275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:54.434251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49167 (local address 127.5.83.126:40065)
0923 20:20:54.435130 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:54.435141 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:54.435183 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:20:54.437058 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:54.437080 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:54.437461 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:54.437785 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:54.439549 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:54.440870 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:54.442676 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:54.443406 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:54.443545 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:54.448644 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:54.448654 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:54.448668 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:54.448708 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:54.451881 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:54.452885 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:54.452892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:54.452898 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:54.453081 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:54.453540 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:54.453549 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:54.453551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:54.454041 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:54.454349 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:54.454825 (+ 476us) server_negotiation.cc:299] Negotiation successful
0923 20:20:54.455141 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":383,"threads_started":1}
I20260923 20:20:54.460136 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:54.463115 27272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d9816d02184809a0510640c7562eef" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40065 } }
I20260923 20:20:54.463497 27272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:54.463613 27272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d9816d02184809a0510640c7562eef, State: Initialized, Role: FOLLOWER
I20260923 20:20:54.464226 27272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d9816d02184809a0510640c7562eef" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40065 } }
I20260923 20:20:54.464519 27272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:54.464704 27272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:54.464874 27272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:54.467034 27272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d9816d02184809a0510640c7562eef" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40065 } }
I20260923 20:20:54.467545 27272 leader_election.cc:302] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82d9816d02184809a0510640c7562eef; no voters:
I20260923 20:20:54.468266 27272 leader_election.cc:288] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:54.468364 27282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:54.470255 27282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [term 1 LEADER]: Becoming Leader. State: Replica: 82d9816d02184809a0510640c7562eef, State: Running, Role: LEADER
I20260923 20:20:54.470886 27272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:54.470888 27282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d9816d02184809a0510640c7562eef" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40065 } }
I20260923 20:20:54.474129 27284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82d9816d02184809a0510640c7562eef. Latest consensus state: current_term: 1 leader_uuid: "82d9816d02184809a0510640c7562eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d9816d02184809a0510640c7562eef" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40065 } } }
I20260923 20:20:54.474701 27284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:54.474319 27283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82d9816d02184809a0510640c7562eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d9816d02184809a0510640c7562eef" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 40065 } } }
I20260923 20:20:54.477926 27283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:54.479161 27292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:54.484748 27275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:54.464575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42315 (local address 127.5.83.126:40065)
0923 20:20:54.464797 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:54.464803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:54.467747 (+ 2944us) server_negotiation.cc:407] Connection header received
0923 20:20:54.467975 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:54.467981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:54.468061 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:54.468157 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:54.470001 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:54.471027 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:54.472374 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:54.473066 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:54.473214 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:54.476956 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:54.476968 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:54.476976 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:20:54.477031 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:54.481021 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:54.482170 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:54.482175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:54.482179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:54.482315 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:20:54.482886 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:20:54.482891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:20:54.482894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:20:54.483369 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:54.483709 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:54.484172 (+ 463us) server_negotiation.cc:299] Negotiation successful
0923 20:20:54.484371 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 20:20:54.485137 27292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:54.493943 27292 catalog_manager.cc:1347] Generated new cluster ID: 7ade55bdb414482db914b698a31748e9
I20260923 20:20:54.494066 27292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:54.524503 27292 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:54.525949 27292 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:54.534518 27292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82d9816d02184809a0510640c7562eef: Generated new TSK 0
I20260923 20:20:54.535403 27292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:54.541407 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27204
I20260923 20:20:54.542598 27267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:20:54.637409 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 27204
W20260923 20:20:54.656119 27189 connection.cc:535] client connection to 127.5.83.126:40065 recv error: Network error: failed to read from TLS socket (remote: 127.5.83.126:40065): Connection reset by peer (error 104)
2026-09-23T20:20:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2073 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:54 dist-test-slave-vl24 krb5kdc[27304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:54 dist-test-slave-vl24 krb5kdc[27304](info): set up 2 sockets
Sep 23 20:20:54 dist-test-slave-vl24 krb5kdc[27304](info): commencing operation
krb5kdc: starting...
W20260923 20:20:55.762249 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:55 dist-test-slave-vl24 krb5kdc[27304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:20:56.044173 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36663
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:36663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:56.380187 27317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:56.380470 27317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:56.380654 27317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:56.387835 27317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:56.387955 27317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:56.388085 27317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:56.388190 27317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:56.388268 27317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:56.398505 27317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:36663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:56.399875 27317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:56.401218 27317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:56.413252 27322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:56.413209 27323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:56.413172 27325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:56.414180 27317 server_base.cc:1034] running on GCE node
I20260923 20:20:56.414923 27317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:56.416304 27317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:56.417500 27317 hybrid_clock.cc:648] HybridClock initialized: now 1790194856417448 us; error 57 us; skew 500 ppm
I20260923 20:20:56.417809 27317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:56 dist-test-slave-vl24 krb5kdc[27304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194856, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:56.420966 27317 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:20:56.422446 27317 webserver.cc:460] Webserver started at http://127.5.83.126:33191/ using document root <none> and password file <none>
I20260923 20:20:56.423141 27317 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:56.423269 27317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:56.423591 27317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:56.426584 27317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "17f46d599eac44c6ae07e06f2429ede8"
format_stamp: "Formatted at 2026-09-23 20:20:56 on dist-test-slave-vl24"
server_key: "91b2b8196515abac8999543a8e97773d"
server_key_iv: "d89b8378c3535763556cbfda8bc1ba94"
server_key_version: "encryptionkey@0"
I20260923 20:20:56.427357 27317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "17f46d599eac44c6ae07e06f2429ede8"
format_stamp: "Formatted at 2026-09-23 20:20:56 on dist-test-slave-vl24"
server_key: "91b2b8196515abac8999543a8e97773d"
server_key_iv: "d89b8378c3535763556cbfda8bc1ba94"
server_key_version: "encryptionkey@0"
I20260923 20:20:56.432602 27317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:20:56.436079 27332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:56.437548 27317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 20:20:56.437726 27317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "17f46d599eac44c6ae07e06f2429ede8"
format_stamp: "Formatted at 2026-09-23 20:20:56 on dist-test-slave-vl24"
server_key: "91b2b8196515abac8999543a8e97773d"
server_key_iv: "d89b8378c3535763556cbfda8bc1ba94"
server_key_version: "encryptionkey@0"
I20260923 20:20:56.437919 27317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:56.455015 27317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:56.459131 27317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:56.459409 27317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:56.478305 27317 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:36663
I20260923 20:20:56.478354 27384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:36663 every 8 connection(s)
I20260923 20:20:56.479686 27317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:56.482683 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 27317
I20260923 20:20:56.482872 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:56.483381 5453 external_mini_cluster.cc:1428] Setting key bb9892334f3f8186a3b37e10a4bd5d17
I20260923 20:20:56.483922 27385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:56.497172 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:56.485234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38573 (local address 127.5.83.126:36663)
0923 20:20:56.486056 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:56.486068 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:56.486106 (+ 38us) server_negotiation.cc:407] Connection header received
0923 20:20:56.488428 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:56.488449 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:56.488905 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:56.489230 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:56.489784 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:56.491037 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:56.492873 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:56.493485 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:56.493607 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:56.494855 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:56.494861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:56.494874 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:56.495458 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:56.495537 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:56.495561 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:56.495684 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:56.495969 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 20:20:56.496160 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":417,"threads_started":1}
I20260923 20:20:56.497665 27385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8: Bootstrap starting.
I20260923 20:20:56.500789 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:56.500833 27385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:56.502025 27385 log.cc:824] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:56.505336 27385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8: No bootstrap required, opened a new log
I20260923 20:20:56.512422 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:56.505866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49749 (local address 127.5.83.126:36663)
0923 20:20:56.506050 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:56.506059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:56.506478 (+ 419us) server_negotiation.cc:407] Connection header received
0923 20:20:56.506635 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:56.506642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:56.506713 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:56.506827 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:56.507245 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:56.508455 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:56.509763 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:20:56.510470 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:20:56.510591 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:20:56.511635 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:56.511641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:56.511646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:56.511667 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:56.511726 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:56.511745 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:56.511874 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:56.512011 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 20:20:56.512064 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 20:20:56.515095 27385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f46d599eac44c6ae07e06f2429ede8" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36663 } }
I20260923 20:20:56.515551 27385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:56.515673 27385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f46d599eac44c6ae07e06f2429ede8, State: Initialized, Role: FOLLOWER
I20260923 20:20:56.516343 27385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f46d599eac44c6ae07e06f2429ede8" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36663 } }
I20260923 20:20:56.516635 27385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:56.516789 27385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:56.516959 27385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:56.519223 27385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f46d599eac44c6ae07e06f2429ede8" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36663 } }
I20260923 20:20:56.519805 27385 leader_election.cc:302] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17f46d599eac44c6ae07e06f2429ede8; no voters:
I20260923 20:20:56.520498 27385 leader_election.cc:288] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:56.520655 27395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:56.521415 27395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [term 1 LEADER]: Becoming Leader. State: Replica: 17f46d599eac44c6ae07e06f2429ede8, State: Running, Role: LEADER
I20260923 20:20:56.522009 27395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f46d599eac44c6ae07e06f2429ede8" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36663 } }
I20260923 20:20:56.522656 27385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:56.525326 27396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17f46d599eac44c6ae07e06f2429ede8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f46d599eac44c6ae07e06f2429ede8" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36663 } } }
I20260923 20:20:56.525686 27396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:56.526492 27397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17f46d599eac44c6ae07e06f2429ede8. Latest consensus state: current_term: 1 leader_uuid: "17f46d599eac44c6ae07e06f2429ede8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f46d599eac44c6ae07e06f2429ede8" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36663 } } }
I20260923 20:20:56.526862 27397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8 [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:56.528481 27402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:56.534179 27402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:56.543972 27402 catalog_manager.cc:1347] Generated new cluster ID: 95b36111bb98472b842154cfcd572d8d
I20260923 20:20:56.544103 27402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:56.564924 27402 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:56.566231 27402 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:56.580758 27402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17f46d599eac44c6ae07e06f2429ede8: Generated new TSK 0
I20260923 20:20:56.581619 27402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:56.589429 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27317
W20260923 20:20:56.590610 27380 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38573
W20260923 20:20:56.591300 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27317: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:20:56.591446 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 27317
W20260923 20:20:56.610605 27302 connection.cc:535] client connection to 127.5.83.126:36663 recv error: Network error: failed to read from TLS socket (remote: 127.5.83.126:36663): Connection reset by peer (error 104)
2026-09-23T20:20:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1954 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:56 dist-test-slave-vl24 krb5kdc[27416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:56 dist-test-slave-vl24 krb5kdc[27416](info): set up 2 sockets
Sep 23 20:20:56 dist-test-slave-vl24 krb5kdc[27416](info): commencing operation
krb5kdc: starting...
W20260923 20:20:57.706410 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:57 dist-test-slave-vl24 krb5kdc[27416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:20:57.991673 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:43599
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:35855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:43599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:20:58.330637 27429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:20:58.330941 27429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:20:58.331101 27429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:20:58.338467 27429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:20:58.338624 27429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:20:58.338729 27429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:20:58.338832 27429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:20:58.338912 27429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:20:58.349391 27429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:35855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:43599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:43599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:20:58.350769 27429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:20:58.352075 27429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:20:58.364027 27435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:58.363981 27434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:20:58.364602 27437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:20:58.370234 27429 server_base.cc:1034] running on GCE node
I20260923 20:20:58.371207 27429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:20:58.372766 27429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:20:58.374003 27429 hybrid_clock.cc:648] HybridClock initialized: now 1790194858373934 us; error 53 us; skew 500 ppm
I20260923 20:20:58.374344 27429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:20:58 dist-test-slave-vl24 krb5kdc[27416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194858, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:20:58.377557 27429 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:20:58.379027 27429 webserver.cc:460] Webserver started at http://127.5.83.126:45583/ using document root <none> and password file <none>
I20260923 20:20:58.379711 27429 fs_manager.cc:359] Metadata directory not provided
I20260923 20:20:58.379822 27429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:20:58.380066 27429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:20:58.383077 27429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "57ee049e4fc44553a8f5974734a6ea1f"
format_stamp: "Formatted at 2026-09-23 20:20:58 on dist-test-slave-vl24"
server_key: "09ae5b311a623edb2d1e5733f03c301e"
server_key_iv: "fb7e280c6c9f5effcf13787ac2fa9a06"
server_key_version: "encryptionkey@0"
I20260923 20:20:58.383795 27429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "57ee049e4fc44553a8f5974734a6ea1f"
format_stamp: "Formatted at 2026-09-23 20:20:58 on dist-test-slave-vl24"
server_key: "09ae5b311a623edb2d1e5733f03c301e"
server_key_iv: "fb7e280c6c9f5effcf13787ac2fa9a06"
server_key_version: "encryptionkey@0"
I20260923 20:20:58.389038 27429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:20:58.392697 27444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:20:58.394239 27429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:20:58.394450 27429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "57ee049e4fc44553a8f5974734a6ea1f"
format_stamp: "Formatted at 2026-09-23 20:20:58 on dist-test-slave-vl24"
server_key: "09ae5b311a623edb2d1e5733f03c301e"
server_key_iv: "fb7e280c6c9f5effcf13787ac2fa9a06"
server_key_version: "encryptionkey@0"
I20260923 20:20:58.394666 27429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:20:58.400624 27429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:20:58.404845 27429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:20:58.405128 27429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:20:58.425029 27429 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:43599
I20260923 20:20:58.425117 27496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:43599 every 8 connection(s)
I20260923 20:20:58.426707 27429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:20:58.430255 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 27429
I20260923 20:20:58.430438 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:20:58.431002 5453 external_mini_cluster.cc:1428] Setting key 2384711b304814f107347d19da161a34
I20260923 20:20:58.431206 27497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:20:58.439354 27500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:58.432878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39071 (local address 127.5.83.126:43599)
0923 20:20:58.433706 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:58.433717 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:58.433761 (+ 44us) server_negotiation.cc:407] Connection header received
0923 20:20:58.435863 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:58.435885 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:58.436233 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:58.436636 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:58.437652 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:58.437661 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:58.437673 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:58.437687 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:58.437747 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:58.437770 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:58.437958 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:58.438061 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 20:20:58.438263 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":396,"threads_started":1}
I20260923 20:20:58.443120 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:20:58.445498 27497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f: Bootstrap starting.
I20260923 20:20:58.449769 27497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f: Neither blocks nor log segments found. Creating new log.
I20260923 20:20:58.451083 27497 log.cc:824] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f: Log is configured to *not* fsync() on all Append() calls
I20260923 20:20:58.451295 27500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:20:58.448404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53299 (local address 127.5.83.126:43599)
0923 20:20:58.448644 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 20:20:58.448650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:20:58.449178 (+ 528us) server_negotiation.cc:407] Connection header received
0923 20:20:58.449314 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:20:58.449319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:20:58.449386 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:20:58.449507 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:20:58.450581 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:20:58.450587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:20:58.450593 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:20:58.450615 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:20:58.450662 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:20:58.450685 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:20:58.450823 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 20:20:58.450917 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 20:20:58.450975 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 20:20:58.454227 27497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f: No bootstrap required, opened a new log
I20260923 20:20:58.462574 27497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ee049e4fc44553a8f5974734a6ea1f" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43599 } }
I20260923 20:20:58.462966 27497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:20:58.463040 27497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ee049e4fc44553a8f5974734a6ea1f, State: Initialized, Role: FOLLOWER
I20260923 20:20:58.463809 27497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ee049e4fc44553a8f5974734a6ea1f" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43599 } }
I20260923 20:20:58.464042 27497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:20:58.464149 27497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:20:58.464269 27497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:20:58.466492 27497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ee049e4fc44553a8f5974734a6ea1f" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43599 } }
I20260923 20:20:58.466960 27497 leader_election.cc:302] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57ee049e4fc44553a8f5974734a6ea1f; no voters:
I20260923 20:20:58.467494 27497 leader_election.cc:288] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:20:58.467679 27507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:20:58.468919 27507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [term 1 LEADER]: Becoming Leader. State: Replica: 57ee049e4fc44553a8f5974734a6ea1f, State: Running, Role: LEADER
I20260923 20:20:58.469460 27507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ee049e4fc44553a8f5974734a6ea1f" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43599 } }
I20260923 20:20:58.469775 27497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:20:58.472427 27509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57ee049e4fc44553a8f5974734a6ea1f. Latest consensus state: current_term: 1 leader_uuid: "57ee049e4fc44553a8f5974734a6ea1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ee049e4fc44553a8f5974734a6ea1f" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43599 } } }
I20260923 20:20:58.472402 27508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57ee049e4fc44553a8f5974734a6ea1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ee049e4fc44553a8f5974734a6ea1f" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 43599 } } }
I20260923 20:20:58.473016 27508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:58.473006 27509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f [sys.catalog]: This master's current role is: LEADER
I20260923 20:20:58.475999 27513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:20:58.481877 27513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:20:58.491361 27513 catalog_manager.cc:1347] Generated new cluster ID: 2546d46cf16547a69b292b1ad698c8ba
I20260923 20:20:58.491504 27513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:20:58.513152 27513 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:20:58.513693 27513 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:20:58.521207 27513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57ee049e4fc44553a8f5974734a6ea1f: Generated new TSK 0
I20260923 20:20:58.522248 27513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:20:58.532951 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27429
W20260923 20:20:58.534051 27491 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39071
W20260923 20:20:58.534825 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27429: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:20:58.535022 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 27429
2026-09-23T20:20:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1943 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:20:58 dist-test-slave-vl24 krb5kdc[27528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:20:58 dist-test-slave-vl24 krb5kdc[27528](info): set up 2 sockets
Sep 23 20:20:58 dist-test-slave-vl24 krb5kdc[27528](info): commencing operation
krb5kdc: starting...
W20260923 20:20:59.658756 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:20:59 dist-test-slave-vl24 krb5kdc[27528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:20:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:20:59.951398 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:38675
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:42189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:38675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:00.292659 27541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:00.293044 27541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:00.293203 27541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:00.301031 27541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:00.301208 27541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:00.301332 27541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:00.301437 27541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:00.301524 27541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:00.311956 27541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:42189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:38675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:38675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:00.313472 27541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:00.314872 27541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:00.327577 27546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:00.327634 27547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:00.327632 27549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:00.328667 27541 server_base.cc:1034] running on GCE node
I20260923 20:21:00.329564 27541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:00.331090 27541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:00.332377 27541 hybrid_clock.cc:648] HybridClock initialized: now 1790194860332329 us; error 42 us; skew 500 ppm
I20260923 20:21:00.332829 27541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:00 dist-test-slave-vl24 krb5kdc[27528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194860, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:00.336220 27541 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:21:00.338444 27541 webserver.cc:460] Webserver started at http://127.5.83.126:34473/ using document root <none> and password file <none>
I20260923 20:21:00.339200 27541 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:00.339360 27541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:00.339707 27541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:00.342877 27541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "3cac597a15344c60993bcf27efd02f96"
format_stamp: "Formatted at 2026-09-23 20:21:00 on dist-test-slave-vl24"
server_key: "3b7a4338cf83f7768f8f5c2766c31f5b"
server_key_iv: "a32996087e4757ca6bd2fe7a22c466db"
server_key_version: "encryptionkey@0"
I20260923 20:21:00.343766 27541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "3cac597a15344c60993bcf27efd02f96"
format_stamp: "Formatted at 2026-09-23 20:21:00 on dist-test-slave-vl24"
server_key: "3b7a4338cf83f7768f8f5c2766c31f5b"
server_key_iv: "a32996087e4757ca6bd2fe7a22c466db"
server_key_version: "encryptionkey@0"
I20260923 20:21:00.349804 27541 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 20:21:00.353317 27556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:00.354686 27541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:21:00.354885 27541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "3cac597a15344c60993bcf27efd02f96"
format_stamp: "Formatted at 2026-09-23 20:21:00 on dist-test-slave-vl24"
server_key: "3b7a4338cf83f7768f8f5c2766c31f5b"
server_key_iv: "a32996087e4757ca6bd2fe7a22c466db"
server_key_version: "encryptionkey@0"
I20260923 20:21:00.355090 27541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:00.376303 27541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:00.381031 27541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:00.381356 27541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:00.400974 27541 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:38675
I20260923 20:21:00.401013 27607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:38675 every 8 connection(s)
I20260923 20:21:00.402482 27541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:00.406342 27608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:00.409922 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 27541
I20260923 20:21:00.410120 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:21:00.410688 5453 external_mini_cluster.cc:1428] Setting key 11506912e5a9dd5ca5a5760d4ce93571
I20260923 20:21:00.418574 27608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96: Bootstrap starting.
I20260923 20:21:00.421772 27608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:00.423130 27608 log.cc:824] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96: Log is configured to *not* fsync() on all Append() calls
Sep 23 20:21:00 dist-test-slave-vl24 krb5kdc[27528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:21:00.426719 27608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96: No bootstrap required, opened a new log
I20260923 20:21:00.433909 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:00.412671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50789 (local address 127.5.83.126:38675)
0923 20:21:00.413398 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:00.413410 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:00.413452 (+ 42us) server_negotiation.cc:407] Connection header received
0923 20:21:00.415136 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:00.415158 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:00.415569 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:00.416023 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:00.417540 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:00.418925 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:00.420216 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:00.420961 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:00.421155 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:00.426372 (+ 5217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:00.426380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:00.426394 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:00.426434 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:00.429676 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:00.430500 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:00.430506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:00.430513 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:00.430674 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:00.431111 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:00.431114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:00.431116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:00.431650 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:00.431863 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:00.432379 (+ 516us) server_negotiation.cc:299] Negotiation successful
0923 20:21:00.432853 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":317,"threads_started":1}
I20260923 20:21:00.435593 27608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cac597a15344c60993bcf27efd02f96" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38675 } }
I20260923 20:21:00.435961 27608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:00.436115 27608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cac597a15344c60993bcf27efd02f96, State: Initialized, Role: FOLLOWER
I20260923 20:21:00.436800 27608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cac597a15344c60993bcf27efd02f96" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38675 } }
I20260923 20:21:00.437055 27608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:21:00.437213 27608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:21:00.437404 27608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:00.437659 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:21:00.440641 27608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cac597a15344c60993bcf27efd02f96" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38675 } }
I20260923 20:21:00.441164 27608 leader_election.cc:302] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cac597a15344c60993bcf27efd02f96; no voters:
I20260923 20:21:00.441855 27608 leader_election.cc:288] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:21:00.441964 27615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:00.443055 27615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [term 1 LEADER]: Becoming Leader. State: Replica: 3cac597a15344c60993bcf27efd02f96, State: Running, Role: LEADER
I20260923 20:21:00.444145 27615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cac597a15344c60993bcf27efd02f96" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38675 } }
I20260923 20:21:00.444370 27608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:00.451149 27619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cac597a15344c60993bcf27efd02f96. Latest consensus state: current_term: 1 leader_uuid: "3cac597a15344c60993bcf27efd02f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cac597a15344c60993bcf27efd02f96" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38675 } } }
I20260923 20:21:00.451659 27619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:00.451130 27618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cac597a15344c60993bcf27efd02f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cac597a15344c60993bcf27efd02f96" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 38675 } } }
I20260923 20:21:00.452374 27618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:00.457589 27628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:00.462963 27628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:00.467202 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:00.443592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39909 (local address 127.5.83.126:38675)
0923 20:21:00.443939 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:00.443948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:00.444747 (+ 799us) server_negotiation.cc:407] Connection header received
0923 20:21:00.444957 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:00.444962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:00.445039 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:00.445154 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:00.447002 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:00.447806 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:00.449331 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:00.450147 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:00.450339 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:00.454121 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:00.454131 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:00.454139 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:00.454193 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:00.461952 (+ 7759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:00.462669 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:00.462679 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:00.462683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:00.462880 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:00.463428 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:00.463437 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:00.463443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:00.463972 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:00.464297 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:00.466267 (+ 1970us) server_negotiation.cc:299] Negotiation successful
0923 20:21:00.466599 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":170}
I20260923 20:21:00.479019 27628 catalog_manager.cc:1347] Generated new cluster ID: 0d48e940980341b8a15ce929320d8e2e
I20260923 20:21:00.479153 27628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:00.489260 27628 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:00.490715 27628 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:00.499601 27628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cac597a15344c60993bcf27efd02f96: Generated new TSK 0
I20260923 20:21:00.500439 27628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:21:00.515594 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27541
I20260923 20:21:00.516781 27603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:00.608366 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 27541
2026-09-23T20:21:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2073 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:00 dist-test-slave-vl24 krb5kdc[27640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:00 dist-test-slave-vl24 krb5kdc[27640](info): set up 2 sockets
Sep 23 20:21:00 dist-test-slave-vl24 krb5kdc[27640](info): commencing operation
krb5kdc: starting...
W20260923 20:21:01.734287 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:21:01 dist-test-slave-vl24 krb5kdc[27640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:21:02.027799 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36051
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:44975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:36051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:02.371940 27653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:02.372227 27653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:02.372421 27653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:02.379786 27653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:02.379913 27653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:02.380003 27653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:02.380118 27653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:02.380184 27653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:02.390065 27653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:44975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:36051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:02.391399 27653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:02.392748 27653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:02.404768 27658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:02.404881 27661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:02.405022 27659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:02.406031 27653 server_base.cc:1034] running on GCE node
I20260923 20:21:02.406881 27653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:02.408314 27653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:02.409526 27653 hybrid_clock.cc:648] HybridClock initialized: now 1790194862409479 us; error 41 us; skew 500 ppm
I20260923 20:21:02.409906 27653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:02 dist-test-slave-vl24 krb5kdc[27640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194862, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:02.413151 27653 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:21:02.414779 27653 webserver.cc:460] Webserver started at http://127.5.83.126:39801/ using document root <none> and password file <none>
I20260923 20:21:02.415483 27653 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:02.415628 27653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:02.415961 27653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:02.419080 27653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "48ed2d8d22b4418ca31864a966fa528c"
format_stamp: "Formatted at 2026-09-23 20:21:02 on dist-test-slave-vl24"
server_key: "405fb0102e516f9ab11834e162298bb7"
server_key_iv: "7b61c0e1a7950b199c4a68aac59b50d5"
server_key_version: "encryptionkey@0"
I20260923 20:21:02.419866 27653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "48ed2d8d22b4418ca31864a966fa528c"
format_stamp: "Formatted at 2026-09-23 20:21:02 on dist-test-slave-vl24"
server_key: "405fb0102e516f9ab11834e162298bb7"
server_key_iv: "7b61c0e1a7950b199c4a68aac59b50d5"
server_key_version: "encryptionkey@0"
I20260923 20:21:02.425218 27653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 20:21:02.428682 27668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:02.430086 27653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:21:02.430289 27653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "48ed2d8d22b4418ca31864a966fa528c"
format_stamp: "Formatted at 2026-09-23 20:21:02 on dist-test-slave-vl24"
server_key: "405fb0102e516f9ab11834e162298bb7"
server_key_iv: "7b61c0e1a7950b199c4a68aac59b50d5"
server_key_version: "encryptionkey@0"
I20260923 20:21:02.430498 27653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:02.449488 27653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:02.453433 27653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:02.453715 27653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:02.473135 27653 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:36051
I20260923 20:21:02.473166 27720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:36051 every 8 connection(s)
I20260923 20:21:02.474517 27653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:02.476033 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 27653
I20260923 20:21:02.476215 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:21:02.476828 5453 external_mini_cluster.cc:1428] Setting key 6a759a3a047b45b09b321ecb4803a19d
I20260923 20:21:02.479115 27721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:02.490128 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:02.478640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49133 (local address 127.5.83.126:36051)
0923 20:21:02.479474 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:02.479486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:02.479533 (+ 47us) server_negotiation.cc:407] Connection header received
0923 20:21:02.481255 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:02.481277 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:02.481674 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:02.482023 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:02.482626 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:02.483956 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:02.485785 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:02.486424 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:02.486547 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:02.487810 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:02.487817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:02.487834 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:21:02.488393 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:02.488469 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:21:02.488495 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:02.488671 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:02.488860 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 20:21:02.489151 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":396,"threads_started":1}
I20260923 20:21:02.491667 27721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c: Bootstrap starting.
I20260923 20:21:02.493906 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:21:02.494962 27721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:02.495937 27721 log.cc:824] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:02.498817 27721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c: No bootstrap required, opened a new log
I20260923 20:21:02.504365 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:02.498307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59729 (local address 127.5.83.126:36051)
0923 20:21:02.498499 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:02.498507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:02.498928 (+ 421us) server_negotiation.cc:407] Connection header received
0923 20:21:02.499135 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:02.499139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:02.499197 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:02.499305 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:02.499816 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:02.500601 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:02.501937 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:02.502377 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:02.502519 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:02.503616 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:02.503621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:02.503624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:21:02.503639 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:02.503677 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:21:02.503692 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:02.503801 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:02.503963 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 20:21:02.504053 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:21:02.506876 27721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ed2d8d22b4418ca31864a966fa528c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36051 } }
I20260923 20:21:02.507259 27721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:02.507364 27721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ed2d8d22b4418ca31864a966fa528c, State: Initialized, Role: FOLLOWER
I20260923 20:21:02.507952 27721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ed2d8d22b4418ca31864a966fa528c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36051 } }
I20260923 20:21:02.508191 27721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:21:02.508328 27721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:21:02.508512 27721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:02.510984 27721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ed2d8d22b4418ca31864a966fa528c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36051 } }
I20260923 20:21:02.511487 27721 leader_election.cc:302] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48ed2d8d22b4418ca31864a966fa528c; no voters:
I20260923 20:21:02.512032 27721 leader_election.cc:288] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:21:02.512166 27731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:02.512925 27731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [term 1 LEADER]: Becoming Leader. State: Replica: 48ed2d8d22b4418ca31864a966fa528c, State: Running, Role: LEADER
I20260923 20:21:02.513429 27731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ed2d8d22b4418ca31864a966fa528c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36051 } }
I20260923 20:21:02.514094 27721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:02.516374 27732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48ed2d8d22b4418ca31864a966fa528c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ed2d8d22b4418ca31864a966fa528c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36051 } } }
I20260923 20:21:02.516952 27732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:02.517417 27733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48ed2d8d22b4418ca31864a966fa528c. Latest consensus state: current_term: 1 leader_uuid: "48ed2d8d22b4418ca31864a966fa528c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ed2d8d22b4418ca31864a966fa528c" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36051 } } }
I20260923 20:21:02.517791 27733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:02.521525 27737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:02.526885 27737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:02.536317 27737 catalog_manager.cc:1347] Generated new cluster ID: 93a0f33e65324e9195a5a2623ebf6e7b
I20260923 20:21:02.536441 27737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:02.555226 27737 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:02.556650 27737 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:02.564455 27737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48ed2d8d22b4418ca31864a966fa528c: Generated new TSK 0
I20260923 20:21:02.565328 27737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:21:02.571673 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27653
W20260923 20:21:02.572870 27716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49133
W20260923 20:21:02.573637 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27653: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:21:02.573789 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 27653
2026-09-23T20:21:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:02 dist-test-slave-vl24 krb5kdc[27752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:02 dist-test-slave-vl24 krb5kdc[27752](info): set up 2 sockets
Sep 23 20:21:02 dist-test-slave-vl24 krb5kdc[27752](info): commencing operation
krb5kdc: starting...
W20260923 20:21:03.694192 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:21:03 dist-test-slave-vl24 krb5kdc[27752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:21:03.985893 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:45881
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:45095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:45881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:04.328878 27765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:04.329162 27765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:04.329373 27765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:04.336712 27765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:04.336820 27765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:04.336934 27765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:04.337021 27765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:04.337098 27765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:04.347200 27765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:45095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:45881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:45881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:04.348557 27765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:04.349882 27765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:04.361927 27774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:04.362064 27770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:04.362031 27772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:04.362833 27765 server_base.cc:1034] running on GCE node
I20260923 20:21:04.363598 27765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:04.365005 27765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:04.366230 27765 hybrid_clock.cc:648] HybridClock initialized: now 1790194864366188 us; error 35 us; skew 500 ppm
I20260923 20:21:04.366587 27765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:04 dist-test-slave-vl24 krb5kdc[27752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194864, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:04.369820 27765 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:21:04.371637 27765 webserver.cc:460] Webserver started at http://127.5.83.126:36981/ using document root <none> and password file <none>
I20260923 20:21:04.372329 27765 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:04.372475 27765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:04.372797 27765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:04.375741 27765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "dc101f5da7454234897b8680eddaed49"
format_stamp: "Formatted at 2026-09-23 20:21:04 on dist-test-slave-vl24"
server_key: "7ed2fad03663487eb9a3fdbe439b1301"
server_key_iv: "acf51e4ea0c57819db9b6d6e8bb12b1f"
server_key_version: "encryptionkey@0"
I20260923 20:21:04.376601 27765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "dc101f5da7454234897b8680eddaed49"
format_stamp: "Formatted at 2026-09-23 20:21:04 on dist-test-slave-vl24"
server_key: "7ed2fad03663487eb9a3fdbe439b1301"
server_key_iv: "acf51e4ea0c57819db9b6d6e8bb12b1f"
server_key_version: "encryptionkey@0"
I20260923 20:21:04.381711 27765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:21:04.385071 27781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:04.386592 27765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 20:21:04.386866 27765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "dc101f5da7454234897b8680eddaed49"
format_stamp: "Formatted at 2026-09-23 20:21:04 on dist-test-slave-vl24"
server_key: "7ed2fad03663487eb9a3fdbe439b1301"
server_key_iv: "acf51e4ea0c57819db9b6d6e8bb12b1f"
server_key_version: "encryptionkey@0"
I20260923 20:21:04.387068 27765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:04.392946 27765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:04.397373 27765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:04.397675 27765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:04.415907 27765 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:45881
I20260923 20:21:04.415954 27832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:45881 every 8 connection(s)
I20260923 20:21:04.417407 27765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:04.421229 27833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:04.425367 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 27765
I20260923 20:21:04.425551 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:21:04.425962 5453 external_mini_cluster.cc:1428] Setting key 54f8d0fa1c4962549389d79469b1392b
I20260923 20:21:04.433560 27833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49: Bootstrap starting.
I20260923 20:21:04.435514 27836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:04.427657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60147 (local address 127.5.83.126:45881)
0923 20:21:04.428457 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:04.428472 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:04.428544 (+ 72us) server_negotiation.cc:407] Connection header received
0923 20:21:04.431620 (+ 3076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:04.431650 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:04.432080 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:04.432517 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:04.433537 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:04.433552 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:04.433570 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:21:04.433590 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:04.433661 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:21:04.433695 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:04.433910 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:04.434038 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 20:21:04.434234 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":323,"threads_started":1}
I20260923 20:21:04.437510 27833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:04.438884 27833 log.cc:824] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:04.439314 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:21:04.442153 27833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49: No bootstrap required, opened a new log
I20260923 20:21:04.446969 27836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:04.444429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48481 (local address 127.5.83.126:45881)
0923 20:21:04.444679 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:04.444685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:04.445023 (+ 338us) server_negotiation.cc:407] Connection header received
0923 20:21:04.445199 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:04.445203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:04.445250 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:04.445342 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:04.446306 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:04.446314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:04.446317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 20:21:04.446334 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:04.446370 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 20:21:04.446385 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:04.446485 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:04.446620 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 20:21:04.446664 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 20:21:04.452503 27833 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc101f5da7454234897b8680eddaed49" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 45881 } }
I20260923 20:21:04.452991 27833 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 20:21:04.453142 27833 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc101f5da7454234897b8680eddaed49, State: Initialized, Role: FOLLOWER
I20260923 20:21:04.453836 27833 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc101f5da7454234897b8680eddaed49" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 45881 } }
I20260923 20:21:04.454106 27833 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 20:21:04.454268 27833 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 20:21:04.454468 27833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:04.457041 27833 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc101f5da7454234897b8680eddaed49" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 45881 } }
I20260923 20:21:04.457717 27833 leader_election.cc:302] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc101f5da7454234897b8680eddaed49; no voters:
I20260923 20:21:04.458431 27833 leader_election.cc:288] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 20:21:04.458523 27843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:04.459198 27843 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [term 1 LEADER]: Becoming Leader. State: Replica: dc101f5da7454234897b8680eddaed49, State: Running, Role: LEADER
I20260923 20:21:04.459800 27843 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc101f5da7454234897b8680eddaed49" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 45881 } }
I20260923 20:21:04.460790 27833 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:04.462832 27844 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc101f5da7454234897b8680eddaed49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc101f5da7454234897b8680eddaed49" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 45881 } } }
I20260923 20:21:04.463281 27844 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:04.462805 27845 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc101f5da7454234897b8680eddaed49. Latest consensus state: current_term: 1 leader_uuid: "dc101f5da7454234897b8680eddaed49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc101f5da7454234897b8680eddaed49" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 45881 } } }
I20260923 20:21:04.463742 27845 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:04.467268 27850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:04.473140 27850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:04.482096 27850 catalog_manager.cc:1347] Generated new cluster ID: 630f7eae51f24f71abdeac693d49bc79
I20260923 20:21:04.482211 27850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:04.498816 27850 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:04.499254 27850 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:04.513085 27850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc101f5da7454234897b8680eddaed49: Generated new TSK 0
I20260923 20:21:04.513963 27850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:21:04.528195 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27765
W20260923 20:21:04.529670 27828 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60147
W20260923 20:21:04.530460 5453 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27765: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 20:21:04.530603 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 27765
2026-09-23T20:21:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1956 ms)
[----------] 18 tests from AuthTokenIssuingTest (36207 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:04 dist-test-slave-vl24 krb5kdc[27864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:04 dist-test-slave-vl24 krb5kdc[27864](info): set up 2 sockets
Sep 23 20:21:04 dist-test-slave-vl24 krb5kdc[27864](info): commencing operation
krb5kdc: starting...
W20260923 20:21:05.662359 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:21:05 dist-test-slave-vl24 krb5kdc[27864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:21:05.972921 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36203
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:36203,127.5.83.125:36127,127.5.83.124:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:06.324990 27877 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:21:06.325381 27877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:06.325518 27877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:06.325675 27877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:06.333171 27877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:06.333269 27877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:06.333379 27877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:06.333487 27877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:06.333599 27877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:06.344209 27877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:36203,127.5.83.125:36127,127.5.83.124:45657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:36203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:06.345683 27877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:06.347008 27877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:06.358793 27882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:06.358770 27883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:06.358846 27885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:06.359825 27877 server_base.cc:1034] running on GCE node
I20260923 20:21:06.360873 27877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:06.362387 27877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:06.363652 27877 hybrid_clock.cc:648] HybridClock initialized: now 1790194866363588 us; error 45 us; skew 500 ppm
I20260923 20:21:06.364020 27877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:06 dist-test-slave-vl24 krb5kdc[27864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194866, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:06.367631 27877 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:21:06.369637 27877 webserver.cc:460] Webserver started at http://127.5.83.126:44991/ using document root <none> and password file <none>
I20260923 20:21:06.370453 27877 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:06.370587 27877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:06.371008 27877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:06.374276 27877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "7816d01ddb6340c991fea30c365607e4"
format_stamp: "Formatted at 2026-09-23 20:21:06 on dist-test-slave-vl24"
server_key: "181b41b86c994c1cab3faa21b1ed8834"
server_key_iv: "c82bda36003dd067e5a45faf11deffb7"
server_key_version: "encryptionkey@0"
I20260923 20:21:06.375108 27877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "7816d01ddb6340c991fea30c365607e4"
format_stamp: "Formatted at 2026-09-23 20:21:06 on dist-test-slave-vl24"
server_key: "181b41b86c994c1cab3faa21b1ed8834"
server_key_iv: "c82bda36003dd067e5a45faf11deffb7"
server_key_version: "encryptionkey@0"
I20260923 20:21:06.380491 27877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 20:21:06.384203 27892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:06.385676 27877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:06.385898 27877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "7816d01ddb6340c991fea30c365607e4"
format_stamp: "Formatted at 2026-09-23 20:21:06 on dist-test-slave-vl24"
server_key: "181b41b86c994c1cab3faa21b1ed8834"
server_key_iv: "c82bda36003dd067e5a45faf11deffb7"
server_key_version: "encryptionkey@0"
I20260923 20:21:06.386128 27877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:06.408608 27877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:06.413158 27877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:06.413460 27877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:06.433223 27877 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:36203
I20260923 20:21:06.433274 27944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:36203 every 8 connection(s)
I20260923 20:21:06.434640 27877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:06.438449 27945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:06.442569 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 27877
I20260923 20:21:06.442809 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:21:06.443467 5453 external_mini_cluster.cc:1428] Setting key 32316b9246b366368115800b9bc7a21e
I20260923 20:21:06.445214 27945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:21:06 dist-test-slave-vl24 krb5kdc[27864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194866, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:21:06.470286 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.448436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36203 (local address 127.0.0.1:44776)
0923 20:21:06.449508 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.449570 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:06.451528 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:06.452489 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:06.452493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:06.453121 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:06.453465 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:06.453482 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:06.455527 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:06.455535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:06.456944 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:06.456961 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:06.458379 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:06.458383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:06.458509 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:06.459200 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:06.459227 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:06.461475 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:06.466070 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:06.466080 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:06.466090 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:06.466477 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:06.466960 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:06.466964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:06.466966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:06.467127 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:06.468190 (+ 1063us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:06.468196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:06.468392 (+ 196us) client_negotiation.cc:769] Sending connection context
0923 20:21:06.468663 (+ 271us) client_negotiation.cc:240] Negotiation successful
0923 20:21:06.469101 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":423,"threads_started":1}
I20260923 20:21:06.470321 27948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.448428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.5.83.126:36203)
0923 20:21:06.450205 (+ 1777us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:06.450215 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:06.450279 (+ 64us) server_negotiation.cc:407] Connection header received
0923 20:21:06.451931 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:06.451942 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:06.452254 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:06.452478 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:06.453690 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:06.455375 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:06.457087 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:06.458138 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:06.458321 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:06.461754 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:06.461772 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:06.461791 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:06.461853 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:06.465914 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:06.466629 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:06.466641 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:06.466646 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:06.466849 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:06.467262 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:06.467270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:06.467276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:06.468040 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:06.468271 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:06.468864 (+ 593us) server_negotiation.cc:299] Negotiation successful
0923 20:21:06.469316 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":1538,"spinlock_wait_cycles":36096,"thread_start_us":1276,"threads_started":1}
I20260923 20:21:06.473445 27945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.83.125@KRBTEST.COM; defaulting to no policy
I20260923 20:21:06.474592 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.474194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:34924)
0923 20:21:06.474385 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.474488 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260923 20:21:06.474921 27893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
I20260923 20:21:06.476444 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.476072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:34926)
0923 20:21:06.476208 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.476281 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.125@KRBTEST.COM" created.
W20260923 20:21:06.476693 27945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:36127: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.125.keytab.
Entry for principal kudu/127.5.83.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.125.keytab.
I20260923 20:21:06.511996 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.125:36127
--webserver_interface=127.5.83.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.125.keytab
--principal=kudu/127.5.83.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:36203,127.5.83.125:36127,127.5.83.124:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:21:06.526039 27945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } attempt: 1
I20260923 20:21:06.527238 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.526805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:34938)
0923 20:21:06.527008 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.527097 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 20:21:06.528321 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.527857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:34946)
0923 20:21:06.528039 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.528135 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:21:06.528519 27945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:36127: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
I20260923 20:21:06.596815 27945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } attempt: 2
I20260923 20:21:06.598057 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.597487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:34962)
0923 20:21:06.597759 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.597881 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 20:21:06.599064 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.598652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:34976)
0923 20:21:06.598814 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.598895 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 20:21:06.599279 27945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:36127: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
I20260923 20:21:06.690541 27945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } attempt: 3
I20260923 20:21:06.691941 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.691277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:34978)
0923 20:21:06.691554 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.691701 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260923 20:21:06.693269 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.692730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:34982)
0923 20:21:06.692938 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.693049 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 20:21:06.693480 27945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:36127: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
I20260923 20:21:06.836771 27945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } attempt: 4
I20260923 20:21:06.838007 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.837482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:34998)
0923 20:21:06.837786 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.837884 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260923 20:21:06.839066 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.838734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:35010)
0923 20:21:06.838865 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.838944 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:21:06.839257 27945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:36127: Network error: Client connection negotiation failed: client connection to 127.5.83.125:36127: connect: Connection refused (error 111)
W20260923 20:21:06.856972 27952 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:21:06.857350 27952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:06.857476 27952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:06.857612 27952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:06.865005 27952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:06.865135 27952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:06.865269 27952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:06.865365 27952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:06.865476 27952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:06.875546 27952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:36203,127.5.83.125:36127,127.5.83.124:45657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.125.keytab
--principal=kudu/127.5.83.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.125:36127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.83.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:06.876996 27952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:06.878360 27952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:06.890110 27957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:06.890079 27960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:06.890028 27958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:06.891263 27952 server_base.cc:1034] running on GCE node
I20260923 20:21:06.892071 27952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:06.893540 27952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:06.894752 27952 hybrid_clock.cc:648] HybridClock initialized: now 1790194866894675 us; error 48 us; skew 500 ppm
I20260923 20:21:06.895087 27952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:06 dist-test-slave-vl24 krb5kdc[27864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194866, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:06.898346 27952 init.cc:377] Logged in from keytab as kudu/127.5.83.125@KRBTEST.COM (short username kudu)
I20260923 20:21:06.899794 27952 webserver.cc:460] Webserver started at http://127.5.83.125:40429/ using document root <none> and password file <none>
I20260923 20:21:06.900514 27952 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:06.900676 27952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:06.900959 27952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:06.903764 27952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/data/instance:
uuid: "6eed43bd8be044729be28ed21e677688"
format_stamp: "Formatted at 2026-09-23 20:21:06 on dist-test-slave-vl24"
server_key: "f51894a6cd5daa22cfc11bcc83ece8e0"
server_key_iv: "70a2177eb00534b63f78ff9e823e30d4"
server_key_version: "encryptionkey@0"
I20260923 20:21:06.904517 27952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/wal/instance:
uuid: "6eed43bd8be044729be28ed21e677688"
format_stamp: "Formatted at 2026-09-23 20:21:06 on dist-test-slave-vl24"
server_key: "f51894a6cd5daa22cfc11bcc83ece8e0"
server_key_iv: "70a2177eb00534b63f78ff9e823e30d4"
server_key_version: "encryptionkey@0"
I20260923 20:21:06.909461 27952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 20:21:06.912848 27967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:06.914184 27952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 20:21:06.914386 27952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/wal
uuid: "6eed43bd8be044729be28ed21e677688"
format_stamp: "Formatted at 2026-09-23 20:21:06 on dist-test-slave-vl24"
server_key: "f51894a6cd5daa22cfc11bcc83ece8e0"
server_key_iv: "70a2177eb00534b63f78ff9e823e30d4"
server_key_version: "encryptionkey@0"
I20260923 20:21:06.914614 27952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:06.931217 27952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:06.935649 27952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:06.935940 27952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:06.954943 27952 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.125:36127
I20260923 20:21:06.954967 28018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.125:36127 every 8 connection(s)
I20260923 20:21:06.956322 27952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
I20260923 20:21:06.960148 28019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:06.961632 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 27952
I20260923 20:21:06.961826 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-1/wal/instance
I20260923 20:21:06.962352 5453 external_mini_cluster.cc:1428] Setting key df32be8ce7778008e5eb31e6a9c6c2ca
I20260923 20:21:06.969938 28019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:21:06 dist-test-slave-vl24 krb5kdc[27864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194866, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.125@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.124@KRBTEST.COM; defaulting to no policy
I20260923 20:21:06.994351 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.973419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.5.83.126:36203)
0923 20:21:06.973945 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:06.973952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:06.975272 (+ 1320us) server_negotiation.cc:407] Connection header received
0923 20:21:06.977505 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:06.977514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:06.977619 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:06.977761 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:06.979395 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:06.980621 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:06.982155 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:06.982935 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:06.983083 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:06.986879 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:06.986888 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:06.986896 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:06.986960 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:06.990845 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:06.991581 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:06.991589 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:06.991594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:06.991750 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:06.992183 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:06.992189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:06.992192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:06.992628 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:06.992807 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:06.993663 (+ 856us) server_negotiation.cc:299] Negotiation successful
0923 20:21:06.993893 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":241,"threads_started":1}
I20260923 20:21:06.995373 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.973970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36203 (local address 127.0.0.1:44792)
0923 20:21:06.974829 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.974881 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:06.977190 (+ 2309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:06.978008 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:06.978018 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:06.978548 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:06.979139 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:06.979158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:06.980775 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:06.980779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:06.981993 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:06.982009 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:06.983148 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:06.983153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:06.983364 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:06.984108 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:06.984130 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:06.986632 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:06.990994 (+ 4362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:06.991003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:06.991017 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:06.991457 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:06.991866 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:06.991869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:06.991872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:06.992065 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:06.992806 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:06.992813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:06.993219 (+ 406us) client_negotiation.cc:769] Sending connection context
0923 20:21:06.993528 (+ 309us) client_negotiation.cc:240] Negotiation successful
0923 20:21:06.994048 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":321,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.124@KRBTEST.COM" created.
I20260923 20:21:06.997320 28019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:21:07 dist-test-slave-vl24 krb5kdc[27864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194866, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.125@KRBTEST.COM for kudu/127.5.83.125@KRBTEST.COM
I20260923 20:21:07.012467 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.998111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:35026)
0923 20:21:06.998313 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:06.998338 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:06.998421 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:06.999394 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:06.999398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:06.999831 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:06.999964 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:06.999972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.001465 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.001473 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.002360 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.002370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.003178 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.003183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.003375 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.003996 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:07.004013 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:07.006353 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:07.009898 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.009906 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.009910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.010275 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.010659 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.010666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.010669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.010814 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.011427 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:07.011431 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:07.011572 (+ 141us) client_negotiation.cc:769] Sending connection context
0923 20:21:07.011725 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 20:21:07.012018 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":3}
I20260923 20:21:07.012504 28024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:06.998388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.5.83.125:36127)
0923 20:21:06.999008 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:06.999014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:06.999037 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:21:06.999077 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:06.999083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:06.999254 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:06.999403 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.000130 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.001327 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.002489 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.003035 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.003170 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.006563 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.006573 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.006585 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.006612 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.009757 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.010415 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.010429 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.010431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.010551 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.010914 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.010922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.010925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.011284 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.011497 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.011896 (+ 399us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.012108 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":478,"thread_start_us":343,"threads_started":1}
I20260923 20:21:07.015236 28019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:21:07.016553 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.015888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41612)
0923 20:21:07.016126 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.016297 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 20:21:07.016873 27971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
I20260923 20:21:07.018606 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.018121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41614)
0923 20:21:07.018365 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.018454 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260923 20:21:07.018883 28019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:45657: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.124.keytab.
Entry for principal kudu/127.5.83.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.124.keytab.
I20260923 20:21:07.032143 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.124:45657
--webserver_interface=127.5.83.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.124.keytab
--principal=kudu/127.5.83.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:36203,127.5.83.125:36127,127.5.83.124:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:21:07.068245 28019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } attempt: 1
I20260923 20:21:07.069515 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.068996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41620)
0923 20:21:07.069207 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.069337 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 20:21:07.070499 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.070137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41630)
0923 20:21:07.070296 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.070375 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:21:07.070698 28019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:45657: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
I20260923 20:21:07.138537 27945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } attempt: 5
I20260923 20:21:07.138926 28019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } attempt: 2
I20260923 20:21:07.140493 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.139972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41638)
0923 20:21:07.140180 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.140295 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 20:21:07.141772 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.141324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41654)
0923 20:21:07.141511 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.141602 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 20:21:07.142083 28019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:45657: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Sep 23 20:21:07 dist-test-slave-vl24 krb5kdc[27864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194866, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for kudu/127.5.83.125@KRBTEST.COM
I20260923 20:21:07.152947 28024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.139439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.5.83.125:36127)
0923 20:21:07.139634 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:07.139640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:07.139693 (+ 53us) server_negotiation.cc:407] Connection header received
0923 20:21:07.139937 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:07.139943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:07.140023 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:07.140183 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.141176 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.142226 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.143398 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.143859 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.144004 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.147534 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.147540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.147552 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.147588 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.150294 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.150963 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.150968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.150972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.151100 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.151477 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.151483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.151487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.151783 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.151950 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.152341 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.152592 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 20:21:07.152942 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.139403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:35032)
0923 20:21:07.139614 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.139637 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:07.139782 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:07.140234 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:07.140240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:07.140778 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:07.141032 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.141046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.142363 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.142371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.143250 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.143261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.144021 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.144027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.144246 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.144956 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:07.144975 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:07.147329 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:07.150441 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.150447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.150451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.150837 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.151211 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.151216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.151220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.151361 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.151902 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:07.151907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:07.152070 (+ 163us) client_negotiation.cc:769] Sending connection context
0923 20:21:07.152233 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 20:21:07.152586 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 20:21:07.154273 27945 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:21:07.155488 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.155113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41668)
0923 20:21:07.155270 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.155344 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 20:21:07.156402 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.156094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41682)
0923 20:21:07.156225 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.156297 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 20:21:07.156665 27945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:45657: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
I20260923 20:21:07.207901 27945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } attempt: 1
I20260923 20:21:07.209200 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.208641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41688)
0923 20:21:07.208873 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.208990 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 20:21:07.210222 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.209848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41704)
0923 20:21:07.209999 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.210090 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:21:07.210475 27945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:45657: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
I20260923 20:21:07.233299 28019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } attempt: 3
I20260923 20:21:07.234556 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.234085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41720)
0923 20:21:07.234299 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.234398 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 20:21:07.235566 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.235178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41732)
0923 20:21:07.235380 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.235464 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260923 20:21:07.235754 28019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:45657: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
I20260923 20:21:07.278734 27945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } attempt: 2
I20260923 20:21:07.280074 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.279426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41744)
0923 20:21:07.279704 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.279850 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:21:07.281212 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.280772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41756)
0923 20:21:07.280939 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.281041 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 20:21:07.281433 27945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:45657: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
W20260923 20:21:07.376981 28027 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:21:07.377328 28027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:07.377444 28027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:07.377574 28027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 20:21:07.378973 28019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } attempt: 4
I20260923 20:21:07.380095 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.379620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41770)
0923 20:21:07.379845 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.379941 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 20:21:07.381063 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.380744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41776)
0923 20:21:07.380888 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.380967 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:21:07.381287 28019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:45657: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
W20260923 20:21:07.385314 28027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:07.385421 28027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:07.385543 28027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:07.385617 28027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:07.385725 28027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:07.387663 27945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } attempt: 3
I20260923 20:21:07.388832 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.388252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41786)
0923 20:21:07.388469 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.388594 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 20:21:07.389947 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.389472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41796)
0923 20:21:07.389681 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.389817 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260923 20:21:07.390146 27945 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:45657: Network error: Client connection negotiation failed: client connection to 127.5.83.124:45657: connect: Connection refused (error 111)
I20260923 20:21:07.396797 28027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:36203,127.5.83.125:36127,127.5.83.124:45657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.124.keytab
--principal=kudu/127.5.83.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.124:45657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.83.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:07.398214 28027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:07.399565 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:07.415247 28033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:07.415642 28032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:07.425220 28035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:07.426401 28027 server_base.cc:1034] running on GCE node
I20260923 20:21:07.427244 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:07.428774 28027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:07.430068 28027 hybrid_clock.cc:648] HybridClock initialized: now 1790194867429977 us; error 63 us; skew 500 ppm
I20260923 20:21:07.430429 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:07 dist-test-slave-vl24 krb5kdc[27864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:07.433624 28027 init.cc:377] Logged in from keytab as kudu/127.5.83.124@KRBTEST.COM (short username kudu)
I20260923 20:21:07.435155 28027 webserver.cc:460] Webserver started at http://127.5.83.124:33879/ using document root <none> and password file <none>
I20260923 20:21:07.435839 28027 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:07.435956 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:07.436280 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:07.439379 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/data/instance:
uuid: "b4800c236f52406d9066ef5e6be22091"
format_stamp: "Formatted at 2026-09-23 20:21:07 on dist-test-slave-vl24"
server_key: "8430efb41d797df57a6477d42c9ffede"
server_key_iv: "e367afb6c12c670bde54c19eb3db83b9"
server_key_version: "encryptionkey@0"
I20260923 20:21:07.440267 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/wal/instance:
uuid: "b4800c236f52406d9066ef5e6be22091"
format_stamp: "Formatted at 2026-09-23 20:21:07 on dist-test-slave-vl24"
server_key: "8430efb41d797df57a6477d42c9ffede"
server_key_iv: "e367afb6c12c670bde54c19eb3db83b9"
server_key_version: "encryptionkey@0"
I20260923 20:21:07.445734 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 20:21:07.449328 28043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:07.450810 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 20:21:07.450994 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/wal
uuid: "b4800c236f52406d9066ef5e6be22091"
format_stamp: "Formatted at 2026-09-23 20:21:07 on dist-test-slave-vl24"
server_key: "8430efb41d797df57a6477d42c9ffede"
server_key_iv: "e367afb6c12c670bde54c19eb3db83b9"
server_key_version: "encryptionkey@0"
I20260923 20:21:07.451205 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:07.462963 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:07.467036 28027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:07.467295 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:07.487221 28027 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.124:45657
I20260923 20:21:07.487263 28094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.124:45657 every 8 connection(s)
I20260923 20:21:07.488703 28027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
I20260923 20:21:07.491110 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 28027
I20260923 20:21:07.491295 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790194506409421-5453-0/minicluster-data/master-2/wal/instance
I20260923 20:21:07.491743 5453 external_mini_cluster.cc:1428] Setting key ae1ac59e375357df504e5dfe06b5d4f4
I20260923 20:21:07.494191 28095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:07.506551 28095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:21:07 dist-test-slave-vl24 krb5kdc[27864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.124@KRBTEST.COM
I20260923 20:21:07.533252 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.493871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.5.83.124:45657)
0923 20:21:07.505396 (+ 11525us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:07.505412 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:07.505907 (+ 495us) server_negotiation.cc:407] Connection header received
0923 20:21:07.507621 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:07.507643 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:07.508062 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:07.508471 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.509947 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.511494 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.514043 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.515493 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.515722 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.522888 (+ 7166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.522903 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.522930 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.523004 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.528449 (+ 5445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.529558 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.529565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.529574 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.529810 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.530270 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.530276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.530279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.530861 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.531203 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.531676 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.532141 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11179,"thread_start_us":10906,"threads_started":1}
Sep 23 20:21:07 dist-test-slave-vl24 krb5kdc[27864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.124@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:21:07.552285 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.511285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:36203 (local address 127.0.0.1:44796)
0923 20:21:07.529193 (+ 17908us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.529264 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:07.529458 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:07.529920 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:07.529931 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:07.530955 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:07.531428 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.531446 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.533494 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.533501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.535703 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.535723 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.537049 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.537059 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.537341 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.538245 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:07.538280 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:07.541785 (+ 3505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:07.548354 (+ 6569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.548400 (+ 46us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.548413 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.549375 (+ 962us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.549991 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.549997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.550003 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.550189 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.550939 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:07.550951 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:07.551190 (+ 239us) client_negotiation.cc:769] Sending connection context
0923 20:21:07.551407 (+ 217us) client_negotiation.cc:240] Negotiation successful
0923 20:21:07.551801 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17547,"thread_start_us":17133,"threads_started":1}
I20260923 20:21:07.554272 28100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.511398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.5.83.126:36203)
0923 20:21:07.527846 (+ 16448us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:07.527855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:07.529388 (+ 1533us) server_negotiation.cc:407] Connection header received
0923 20:21:07.529602 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:07.529608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:07.529722 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:07.529840 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.531674 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.533234 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.535913 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.536589 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.536750 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.542078 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.542089 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.542098 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.542153 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.548090 (+ 5937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.549614 (+ 1524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.549621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.549626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.549818 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.550345 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.550352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.550358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.550771 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.553234 (+ 2463us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.553387 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.553667 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16262,"thread_start_us":16470,"threads_started":1}
Sep 23 20:21:07 dist-test-slave-vl24 krb5kdc[27864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:21:07.556344 28095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:21:07 dist-test-slave-vl24 krb5kdc[27864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.124@KRBTEST.COM for kudu/127.5.83.125@KRBTEST.COM
I20260923 20:21:07.565219 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.536382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44812 (local address 127.5.83.126:36203)
0923 20:21:07.542562 (+ 6180us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:07.542571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:07.542598 (+ 27us) server_negotiation.cc:407] Connection header received
0923 20:21:07.542664 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:07.542671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:07.542803 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:07.542926 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.544191 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.545902 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.548291 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.549233 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.549378 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.557712 (+ 8334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.557720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.557725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.557766 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.561248 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.562233 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.562238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.562242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.562424 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.562964 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.562973 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.562978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.563448 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.563808 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.564135 (+ 327us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.564413 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5979,"thread_start_us":5725,"threads_started":1}
I20260923 20:21:07.570039 27945 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } attempt: 4
I20260923 20:21:07.575394 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.557290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:36127 (local address 127.0.0.1:35034)
0923 20:21:07.557591 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.557612 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:07.557736 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:07.558155 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:07.558160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:07.558574 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:07.558817 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.558831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.560618 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.560624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.561821 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.561832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.562731 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.562737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.562912 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.563475 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:07.563492 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:07.566006 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:07.571109 (+ 5103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.571118 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.571124 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.572236 (+ 1112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.572822 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.572829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.572835 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.573068 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.574205 (+ 1137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:07.574213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:07.574400 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 20:21:07.574569 (+ 169us) client_negotiation.cc:240] Negotiation successful
0923 20:21:07.574925 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260923 20:21:07.576704 28024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.557370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.5.83.125:36127)
0923 20:21:07.557605 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:07.557614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:07.557673 (+ 59us) server_negotiation.cc:407] Connection header received
0923 20:21:07.557881 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:07.557890 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:07.557991 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:07.558117 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.559010 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.560394 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.561971 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.562573 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.562718 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.566227 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.566236 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.566243 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.566291 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.570913 (+ 4622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.572400 (+ 1487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.572408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.572414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.572661 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.573211 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.573218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.573224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.573958 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.574133 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.575897 (+ 1764us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.576163 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Sep 23 20:21:07 dist-test-slave-vl24 krb5kdc[27864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.125@KRBTEST.COM
I20260923 20:21:07.578668 28095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:21:07 dist-test-slave-vl24 krb5kdc[27864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194866, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for kudu/127.5.83.124@KRBTEST.COM
I20260923 20:21:07.586795 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.567013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.5.83.125:36127)
0923 20:21:07.567674 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:07.567679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:07.567697 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:21:07.567741 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:07.567745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:07.567798 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:07.567930 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.569254 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.570184 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.571856 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.572580 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.572716 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.578361 (+ 5645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.578369 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.578376 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.578427 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.582768 (+ 4341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.583649 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.583657 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.583663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.583817 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.584317 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.584324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.584327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.584730 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.584992 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.586107 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.586330 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":287,"threads_started":1}
I20260923 20:21:07.588928 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.571144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41808)
0923 20:21:07.571345 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.571368 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:07.571499 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:07.571992 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:07.571997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:07.572478 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:07.572712 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.572725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.574048 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.574053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.575749 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.575766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.576978 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.576984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.577198 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.577779 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:07.577804 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:07.581637 (+ 3833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:07.585755 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.585760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.585764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.586303 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.586720 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.586726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.586733 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.587177 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.587769 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:07.587774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:07.587985 (+ 211us) client_negotiation.cc:769] Sending connection context
0923 20:21:07.588128 (+ 143us) client_negotiation.cc:240] Negotiation successful
0923 20:21:07.588457 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 20:21:07.589807 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.570811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.5.83.124:45657)
0923 20:21:07.570995 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:07.571001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:07.571457 (+ 456us) server_negotiation.cc:407] Connection header received
0923 20:21:07.571624 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:07.571630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:07.571709 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:07.571800 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.572874 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.573875 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.575937 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.576661 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.576785 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.581845 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.581851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.581855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.581891 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.585576 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.586441 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.586447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.586451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.586580 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.587311 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.587318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.587323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.587631 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.587794 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.589215 (+ 1421us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.589465 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 20:21:07 dist-test-slave-vl24 krb5kdc[27864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194867, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.124@KRBTEST.COM for kudu/127.5.83.124@KRBTEST.COM
I20260923 20:21:07.593755 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:21:07.599042 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.579916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41814)
0923 20:21:07.580125 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.580145 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:07.580250 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:07.581404 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:07.581410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:07.581980 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:07.582221 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.582239 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.584354 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.584363 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.586137 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.586156 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.588094 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.588103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.588336 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.590383 (+ 2047us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:07.590407 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:07.592759 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:07.596475 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.596480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.596483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.597029 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.597386 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.597392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.597395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.597519 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.598025 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:07.598029 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:07.598161 (+ 132us) client_negotiation.cc:769] Sending connection context
0923 20:21:07.598298 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 20:21:07.598617 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 20:21:07.599153 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.580380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.5.83.124:45657)
0923 20:21:07.581078 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:07.581086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:07.581111 (+ 25us) server_negotiation.cc:407] Connection header received
0923 20:21:07.581160 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:07.581167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:07.581244 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:07.582377 (+ 1133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.582643 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.584194 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.586333 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.587165 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.588662 (+ 1497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.592983 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.592992 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.592999 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.593038 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.596312 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.597162 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.597167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.597171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.597279 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.597628 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.597633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.597636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.597911 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.598061 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.598403 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.598747 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":554,"thread_start_us":363,"threads_started":1}
I20260923 20:21:07.600273 27945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4: Bootstrap starting.
I20260923 20:21:07.603498 28095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091: Bootstrap starting.
I20260923 20:21:07.605154 27945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:07.606904 27945 log.cc:824] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:07.607278 28095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:07.608584 28095 log.cc:824] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:07.604198 27920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7816d01ddb6340c991fea30c365607e4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44812
I20260923 20:21:07.613785 27945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4: No bootstrap required, opened a new log
I20260923 20:21:07.613970 28095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091: No bootstrap required, opened a new log
I20260923 20:21:07.623924 27945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } }
I20260923 20:21:07.624574 27945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7816d01ddb6340c991fea30c365607e4, State: Initialized, Role: FOLLOWER
I20260923 20:21:07.624805 28095 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } }
I20260923 20:21:07.625226 28095 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4800c236f52406d9066ef5e6be22091, State: Initialized, Role: FOLLOWER
I20260923 20:21:07.625186 27945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } }
I20260923 20:21:07.625671 28095 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } }
I20260923 20:21:07.626750 28106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } } }
I20260923 20:21:07.627882 27945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:07.628084 27920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 20:21:07.628340 27920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:07.627779 28106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:07.632638 27920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } }
I20260923 20:21:07.633114 28107 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } } }
I20260923 20:21:07.633548 28107 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:07.634198 28095 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:07.636725 28070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7816d01ddb6340c991fea30c365607e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4800c236f52406d9066ef5e6be22091"
I20260923 20:21:07.637301 28070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:07.641770 28070 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7816d01ddb6340c991fea30c365607e4 in term 1.
I20260923 20:21:07.642617 27896 leader_election.cc:302] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7816d01ddb6340c991fea30c365607e4, b4800c236f52406d9066ef5e6be22091; no voters:
I20260923 20:21:07.643214 28106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:07.647848 27920 leader_election.cc:288] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [CANDIDATE]: Term 1 election: Requested vote from peers 6eed43bd8be044729be28ed21e677688 (127.5.83.125:36127), b4800c236f52406d9066ef5e6be22091 (127.5.83.124:45657)
W20260923 20:21:07.666450 27898 tablet.cc:2369] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:21:07.673913 28106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7816d01ddb6340c991fea30c365607e4, State: Running, Role: LEADER
I20260923 20:21:07.680495 28019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } attempt: 5
I20260923 20:21:07.681206 28106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } }
Sep 23 20:21:07 dist-test-slave-vl24 krb5kdc[27864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194866, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.125@KRBTEST.COM for kudu/127.5.83.124@KRBTEST.COM
I20260923 20:21:07.696223 28112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7816d01ddb6340c991fea30c365607e4. Latest consensus state: current_term: 1 leader_uuid: "7816d01ddb6340c991fea30c365607e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } } }
I20260923 20:21:07.697148 28112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:07.707880 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.681151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:45657 (local address 127.0.0.1:41826)
0923 20:21:07.681662 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:07.681682 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:07.681809 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:07.683208 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:07.683212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:07.683742 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:07.683997 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.684010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.685954 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.685959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.687516 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:07.687532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.688770 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.688779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:07.689064 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.689844 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:07.689875 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:07.693114 (+ 3239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:07.702283 (+ 9169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.702312 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.702316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.703305 (+ 989us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.704232 (+ 927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:07.704264 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:07.704271 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:07.704627 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:07.705875 (+ 1248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:07.705883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:07.706186 (+ 303us) client_negotiation.cc:769] Sending connection context
0923 20:21:07.706435 (+ 249us) client_negotiation.cc:240] Negotiation successful
0923 20:21:07.706987 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361}
I20260923 20:21:07.710434 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.682269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.5.83.124:45657)
0923 20:21:07.682699 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:07.682706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:07.682729 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:21:07.682796 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:07.682801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:07.682886 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:07.683003 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.684506 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.685758 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.687711 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.688394 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.688545 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.693377 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.693387 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.693394 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.693447 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.702080 (+ 8633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.703658 (+ 1578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.703668 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.703674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.704004 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.704894 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.704902 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.704908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.705677 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.708861 (+ 3184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.709120 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.709642 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297}
I20260923 20:21:07.716126 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.682073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.5.83.126:36203)
0923 20:21:07.682542 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:07.682548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:07.683343 (+ 795us) server_negotiation.cc:407] Connection header received
0923 20:21:07.691884 (+ 8541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:07.691892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:07.691984 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:07.692080 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.694354 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.696194 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.698297 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.699276 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.699400 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.708821 (+ 9421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.708829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.708834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.708883 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.712841 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.713726 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.713732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.713735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.713909 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.714373 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.714378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.714382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.714752 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.714985 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.715513 (+ 528us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.715741 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341}
W20260923 20:21:07.719269 28129 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:21:07.719393 28129 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:21:07.719493 28130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260923 20:21:07.719585 28049 tablet.cc:2369] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:21:07.719774 28019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688: Bootstrap starting.
I20260923 20:21:07.721618 28130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:07.726958 28019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:07.730610 28019 log.cc:824] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:07.733825 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.684251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.5.83.125:36127)
0923 20:21:07.684420 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:07.684427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:07.690458 (+ 6031us) server_negotiation.cc:407] Connection header received
0923 20:21:07.691414 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:07.691420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:07.691544 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:07.691638 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.718277 (+ 26639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.719070 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.721086 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.721927 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.722044 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.725966 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.725974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.725982 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.726029 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.730406 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.731338 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.731343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.731347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.731504 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.731990 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.731996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.732000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.732365 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.732637 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.733213 (+ 576us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.733463 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 20:21:07.739607 28019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688: No bootstrap required, opened a new log
I20260923 20:21:07.702544 27877 master_runner.cc:186] Error getting master registration for 127.5.83.125:36127: OK, instance_id { permanent_uuid: "6eed43bd8be044729be28ed21e677688" instance_seqno: 1790194866949515 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:21:07.741259 27877 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:21:07.749008 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:07.683708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.5.83.124:45657)
0923 20:21:07.683961 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:07.683969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:07.691313 (+ 7344us) server_negotiation.cc:407] Connection header received
0923 20:21:07.692189 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:07.692194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:07.692284 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:07.692363 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:07.701373 (+ 9010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.702441 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.734827 (+ 32386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:07.735575 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:07.735676 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:07.740791 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:07.740799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:07.740804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:07.740849 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:07.744477 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.745876 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.745883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.745887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.746034 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:07.746710 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:07.746716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:07.746719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:07.747130 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:07.747393 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:07.748236 (+ 843us) server_negotiation.cc:299] Negotiation successful
0923 20:21:07.748483 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 20:21:07.755767 28019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } }
I20260923 20:21:07.760754 28019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eed43bd8be044729be28ed21e677688, State: Initialized, Role: FOLLOWER
I20260923 20:21:07.761534 28019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } }
I20260923 20:21:07.764518 28019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:07.764740 27994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7816d01ddb6340c991fea30c365607e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6eed43bd8be044729be28ed21e677688"
I20260923 20:21:07.765282 27994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:07.769719 27994 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7816d01ddb6340c991fea30c365607e4 in term 1.
I20260923 20:21:07.763093 28137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } } }
I20260923 20:21:07.771056 28137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:07.774621 27994 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [term 1 FOLLOWER]: Refusing update from remote peer 7816d01ddb6340c991fea30c365607e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:07.777987 28112 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:07.781518 28070 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [term 1 FOLLOWER]: Refusing update from remote peer 7816d01ddb6340c991fea30c365607e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:07.785717 28106 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:07.836508 28107 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7816d01ddb6340c991fea30c365607e4. Latest consensus state: current_term: 1 leader_uuid: "7816d01ddb6340c991fea30c365607e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } } }
I20260923 20:21:07.836876 28107 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:07.837812 28134 mvcc.cc:205] Tried to move back new op lower bound from 7332638178375626752 to 7332638178044596224. Current Snapshot: MvccSnapshot[applied={T|T < 7332638178375626752}]
I20260923 20:21:07.853053 28141 mvcc.cc:205] Tried to move back new op lower bound from 7332638178375626752 to 7332638178044596224. Current Snapshot: MvccSnapshot[applied={T|T < 7332638178375626752}]
I20260923 20:21:07.861868 28137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7816d01ddb6340c991fea30c365607e4. Latest consensus state: current_term: 1 leader_uuid: "7816d01ddb6340c991fea30c365607e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } } }
I20260923 20:21:07.862330 28137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:07.876041 28107 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7816d01ddb6340c991fea30c365607e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } } }
I20260923 20:21:07.876364 28107 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:07.878610 28130 catalog_manager.cc:1347] Generated new cluster ID: e8be0a80dbd1476a90f9503d01193c5c
I20260923 20:21:07.878787 28130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:07.883731 28137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7816d01ddb6340c991fea30c365607e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } } }
I20260923 20:21:07.884037 28137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 20:21:07.889020 28160 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:21:07.889158 28160 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:21:07.903594 28112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7816d01ddb6340c991fea30c365607e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } } }
I20260923 20:21:07.903961 28112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:07.904386 28112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7816d01ddb6340c991fea30c365607e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7816d01ddb6340c991fea30c365607e4" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 36203 } } peers { permanent_uuid: "6eed43bd8be044729be28ed21e677688" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 36127 } } peers { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 45657 } } }
I20260923 20:21:07.904667 28112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4 [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:07.905383 28158 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 20:21:07.906504 27952 master_runner.cc:186] Error getting master registration for 127.5.83.124:45657: OK, instance_id { permanent_uuid: "b4800c236f52406d9066ef5e6be22091" instance_seqno: 1790194867481569 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:21:07.919927 27952 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:21:07.927295 28130 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:07.929567 28130 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:07.943915 28130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7816d01ddb6340c991fea30c365607e4: Generated new TSK 0
I20260923 20:21:07.945047 28130 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 20:21:08.043867 28102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.029496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.5.83.125:36127)
0923 20:21:08.029723 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.029732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.041271 (+ 11539us) server_negotiation.cc:407] Connection header received
0923 20:21:08.041637 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.041647 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.041779 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.041873 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.043480 (+ 1607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35052: BlockingRecv error: recv got EOF from 127.0.0.1:35052 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260923 20:21:08.047058 28098 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.035610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.5.83.124:45657)
0923 20:21:08.035832 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.035838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.045222 (+ 9384us) server_negotiation.cc:407] Connection header received
0923 20:21:08.045509 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.045516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.045606 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.045683 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.046764 (+ 1081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41846: BlockingRecv error: recv got EOF from 127.0.0.1:41846 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260923 20:21:08.050134 28101 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.029770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.5.83.126:36203)
0923 20:21:08.029987 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.029994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.048092 (+ 18098us) server_negotiation.cc:407] Connection header received
0923 20:21:08.048361 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.048368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.048459 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.048587 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.049811 (+ 1224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44840: BlockingRecv error: recv got EOF from 127.0.0.1:44840 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260923 20:21:08.061914 28102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.059406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.5.83.125:36127)
0923 20:21:08.059606 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.059612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.060092 (+ 480us) server_negotiation.cc:407] Connection header received
0923 20:21:08.060412 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.060420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.060635 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.060773 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.061621 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35064: BlockingRecv error: recv got EOF from 127.0.0.1:35064 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260923 20:21:08.077993 28102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.074950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.5.83.125:36127)
0923 20:21:08.075166 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.075173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.075688 (+ 515us) server_negotiation.cc:407] Connection header received
0923 20:21:08.075916 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.075925 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.076052 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.076189 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.077129 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35080: BlockingRecv error: recv got EOF from 127.0.0.1:35080 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260923 20:21:08.089812 28102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.087089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.5.83.125:36127)
0923 20:21:08.087280 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.087287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.088170 (+ 883us) server_negotiation.cc:407] Connection header received
0923 20:21:08.088372 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.088380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.088488 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.088636 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.089546 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35092: BlockingRecv error: recv got EOF from 127.0.0.1:35092 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 20:21:08.100597 28102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.098345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.5.83.125:36127)
0923 20:21:08.098525 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.098532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.099041 (+ 509us) server_negotiation.cc:407] Connection header received
0923 20:21:08.099248 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.099255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.099355 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.099472 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.100289 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35094: BlockingRecv error: recv got EOF from 127.0.0.1:35094 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 20:21:08.115283 28102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.113034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.5.83.125:36127)
0923 20:21:08.113227 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.113233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.113733 (+ 500us) server_negotiation.cc:407] Connection header received
0923 20:21:08.113971 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.113978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.114092 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.114201 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.115050 (+ 849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35098: BlockingRecv error: recv got EOF from 127.0.0.1:35098 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260923 20:21:08.137189 28102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.134835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.5.83.125:36127)
0923 20:21:08.135036 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.135042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.135604 (+ 562us) server_negotiation.cc:407] Connection header received
0923 20:21:08.135788 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.135794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.135896 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.136015 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.136891 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35106: BlockingRecv error: recv got EOF from 127.0.0.1:35106 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260923 20:21:08.176496 28102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.174230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35114 (local address 127.5.83.125:36127)
0923 20:21:08.174426 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.174433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.174978 (+ 545us) server_negotiation.cc:407] Connection header received
0923 20:21:08.175179 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.175186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.175285 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.175392 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.176231 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35114: BlockingRecv error: recv got EOF from 127.0.0.1:35114 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 20:21:08.259354 28102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.256864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.5.83.125:36127)
0923 20:21:08.257075 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.257082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.257665 (+ 583us) server_negotiation.cc:407] Connection header received
0923 20:21:08.257891 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.257904 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.258030 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.258144 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.259112 (+ 968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35118: BlockingRecv error: recv got EOF from 127.0.0.1:35118 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260923 20:21:08.392825 28102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.390858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.5.83.125:36127)
0923 20:21:08.391096 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.391102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.391345 (+ 243us) server_negotiation.cc:407] Connection header received
0923 20:21:08.391575 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.391580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.391667 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.391778 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.392592 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36122: BlockingRecv error: recv got EOF from 127.0.0.1:36122 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260923 20:21:08.657847 28102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:08.654581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.5.83.125:36127)
0923 20:21:08.654766 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:08.654772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:08.655490 (+ 718us) server_negotiation.cc:407] Connection header received
0923 20:21:08.655696 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:08.655703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:08.655849 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:08.656005 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:08.657446 (+ 1441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36124: BlockingRecv error: recv got EOF from 127.0.0.1:36124 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 20:21:08.891011 28160 catalog_manager.cc:1259] Loaded cluster ID: e8be0a80dbd1476a90f9503d01193c5c
I20260923 20:21:08.891104 28160 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688: loading cluster ID for follower catalog manager: success
I20260923 20:21:08.893501 28160 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688: acquiring CA information for follower catalog manager: success
I20260923 20:21:08.895109 28160 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6eed43bd8be044729be28ed21e677688: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:21:08.909585 28158 catalog_manager.cc:1259] Loaded cluster ID: e8be0a80dbd1476a90f9503d01193c5c
I20260923 20:21:08.909691 28158 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091: loading cluster ID for follower catalog manager: success
I20260923 20:21:08.912215 28158 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091: acquiring CA information for follower catalog manager: success
I20260923 20:21:08.914146 28158 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b4800c236f52406d9066ef5e6be22091: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:21:09.191355 28229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:09.183845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.5.83.125:36127)
0923 20:21:09.184455 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:09.184461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:09.184835 (+ 374us) server_negotiation.cc:407] Connection header received
0923 20:21:09.185066 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:09.185071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:09.185223 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:09.185342 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:09.185928 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:09.186662 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:09.188741 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:09.189193 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:09.189317 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:09.190042 (+ 725us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:09.190664 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:09.190792 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:09.190945 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 20:21:09.191022 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":298,"threads_started":1}
I20260923 20:21:09.221101 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:09.204360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.5.83.124:45657)
0923 20:21:09.214366 (+ 10006us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:09.214371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:09.214394 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:21:09.214498 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:09.214502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:09.214661 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:09.214831 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:09.215388 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:09.216211 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:09.218358 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:09.218917 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:09.219016 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:09.219686 (+ 670us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:09.220270 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:09.220414 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:09.220620 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 20:21:09.220706 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9866,"thread_start_us":9704,"threads_started":1}
I20260923 20:21:09.246804 28229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:09.228746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.5.83.125:36127)
0923 20:21:09.228960 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:09.228969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:09.239447 (+ 10478us) server_negotiation.cc:407] Connection header received
0923 20:21:09.239672 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:09.239682 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:09.239819 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:09.239946 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:09.241047 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:09.242224 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:09.244472 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:09.245208 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:09.245373 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:09.245901 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:09.246058 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:09.246122 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:09.246403 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 20:21:09.246499 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 20:21:09.248679 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:09.228358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42522 (local address 127.5.83.126:36203)
0923 20:21:09.241374 (+ 13016us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:09.241380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:09.241399 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:21:09.241547 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:09.241552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:09.241796 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:09.242614 (+ 818us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:09.242775 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:09.243845 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:09.245628 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:09.246271 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:09.246873 (+ 602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:09.247069 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:09.247560 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:09.248193 (+ 633us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:09.248246 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 20:21:09.248335 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12879,"thread_start_us":16086,"threads_started":1}
I20260923 20:21:09.249495 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:09.235172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.5.83.124:45657)
0923 20:21:09.235388 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:09.235397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:09.240565 (+ 5168us) server_negotiation.cc:407] Connection header received
0923 20:21:09.240811 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:09.240821 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:09.240955 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:09.241043 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:09.241807 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:09.242970 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:09.246452 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:09.247487 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:09.247607 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:09.248364 (+ 757us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:09.248509 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:09.248611 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:09.249101 (+ 490us) server_negotiation.cc:299] Negotiation successful
0923 20:21:09.249198 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 20:21:09.253854 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27877
I20260923 20:21:09.255100 27940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:09.368086 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 27877
W20260923 20:21:09.392201 27969 connection.cc:535] server connection from 127.0.0.1:35032 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 20:21:09.392246 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 27952
I20260923 20:21:09.393553 28014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:09.503137 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 27952
I20260923 20:21:09.524747 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 28027
I20260923 20:21:09.526101 28090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:09.631642 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 28027
2026-09-23T20:21:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5106 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 20:21:09 dist-test-slave-vl24 krb5kdc[28250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 20:21:09 dist-test-slave-vl24 krb5kdc[28250](info): set up 2 sockets
Sep 23 20:21:09 dist-test-slave-vl24 krb5kdc[28250](info): commencing operation
krb5kdc: starting...
W20260923 20:21:10.766512 5453 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/test-user.keytab.
Sep 23 20:21:10 dist-test-slave-vl24 krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T20:21:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T20:21:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.83.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
Entry for principal kudu/127.5.83.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab.
I20260923 20:21:11.065007 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:46099
--webserver_interface=127.5.83.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:33369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:46099,127.5.83.125:34935,127.5.83.124:37389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 20:21:11.415942 28263 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:21:11.416283 28263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:11.416358 28263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:11.416440 28263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:11.423776 28263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:11.423866 28263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:11.423923 28263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:11.423966 28263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:11.424029 28263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:11.434008 28263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:33369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:46099,127.5.83.125:34935,127.5.83.124:37389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.126.keytab
--principal=kudu/127.5.83.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.126:46099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.83.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:11.435274 28263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:11.436594 28263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:11.448568 28268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:11.448649 28269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:11.448639 28271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:11.449266 28263 server_base.cc:1034] running on GCE node
I20260923 20:21:11.450156 28263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:11.451660 28263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:11.452922 28263 hybrid_clock.cc:648] HybridClock initialized: now 1790194871452855 us; error 41 us; skew 500 ppm
I20260923 20:21:11.453290 28263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:11 dist-test-slave-vl24 krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194871, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:11.463016 28263 init.cc:377] Logged in from keytab as kudu/127.5.83.126@KRBTEST.COM (short username kudu)
I20260923 20:21:11.464768 28263 webserver.cc:460] Webserver started at http://127.5.83.126:35767/ using document root <none> and password file <none>
I20260923 20:21:11.465507 28263 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:11.465641 28263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:11.465975 28263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:11.469245 28263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/data/instance:
uuid: "31102c333ecb49059450bae419a179bb"
format_stamp: "Formatted at 2026-09-23 20:21:11 on dist-test-slave-vl24"
server_key: "0e114c899670f9c7d8072561fb877352"
server_key_iv: "7289be18704cad39cd1979a80f94b185"
server_key_version: "encryptionkey@0"
I20260923 20:21:11.470065 28263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/wal/instance:
uuid: "31102c333ecb49059450bae419a179bb"
format_stamp: "Formatted at 2026-09-23 20:21:11 on dist-test-slave-vl24"
server_key: "0e114c899670f9c7d8072561fb877352"
server_key_iv: "7289be18704cad39cd1979a80f94b185"
server_key_version: "encryptionkey@0"
I20260923 20:21:11.475282 28263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 20:21:11.478763 28278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:11.480057 28263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 20:21:11.480255 28263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/wal
uuid: "31102c333ecb49059450bae419a179bb"
format_stamp: "Formatted at 2026-09-23 20:21:11 on dist-test-slave-vl24"
server_key: "0e114c899670f9c7d8072561fb877352"
server_key_iv: "7289be18704cad39cd1979a80f94b185"
server_key_version: "encryptionkey@0"
I20260923 20:21:11.480496 28263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:11.504750 28263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:11.508746 28263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:11.509037 28263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:11.527776 28263 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.126:46099
I20260923 20:21:11.527784 28330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.126:46099 every 8 connection(s)
I20260923 20:21:11.529142 28263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/data/info.pb
I20260923 20:21:11.532819 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:11.534550 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 28263
I20260923 20:21:11.534791 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-0/wal/instance
I20260923 20:21:11.535216 5453 external_mini_cluster.cc:1428] Setting key 243b66a3bc5ad3edf22d0f4bd1ad5978
I20260923 20:21:11.539424 28331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:21:11 dist-test-slave-vl24 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194871, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.125@KRBTEST.COM; defaulting to no policy
I20260923 20:21:11.562028 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:11.542665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46099 (local address 127.0.0.1:47218)
0923 20:21:11.543865 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:11.543898 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:11.545740 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:11.546578 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:11.546585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:11.547297 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:11.547630 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:11.547647 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:11.549295 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:11.549299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:11.550366 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:11.550378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:11.551280 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:11.551287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:11.551467 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:11.552124 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:11.552147 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:11.554358 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:11.557645 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:11.557656 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:11.557668 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:11.558140 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:11.558544 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:11.558551 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:11.558554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:11.558720 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:11.559489 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:11.559498 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:11.559697 (+ 199us) client_negotiation.cc:769] Sending connection context
0923 20:21:11.560002 (+ 305us) client_negotiation.cc:240] Negotiation successful
0923 20:21:11.560670 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":142,"thread_start_us":777,"threads_started":1}
I20260923 20:21:11.562047 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:11.542665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.5.83.126:46099)
0923 20:21:11.543567 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:11.543585 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:11.544038 (+ 453us) server_negotiation.cc:407] Connection header received
0923 20:21:11.546114 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:11.546120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:11.546354 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:11.546569 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:11.547837 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:11.549127 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:11.550505 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:11.551055 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:11.551250 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:11.554584 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:11.554592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:11.554605 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:11.554638 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:11.557504 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:11.558269 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:11.558285 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:11.558287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:11.558436 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:11.558832 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:11.558837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:11.558847 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:11.559333 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:11.559557 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:11.560121 (+ 564us) server_negotiation.cc:299] Negotiation successful
0923 20:21:11.560534 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":634,"thread_start_us":425,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.125@KRBTEST.COM" created.
I20260923 20:21:11.565068 28331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:21:11.566217 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:11.565797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44202)
0923 20:21:11.565974 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:11.566076 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260923 20:21:11.566452 28280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
I20260923 20:21:11.568087 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:11.567674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44206)
0923 20:21:11.567851 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:11.567941 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 20:21:11.568302 28331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:34935: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.125.keytab.
Entry for principal kudu/127.5.83.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.125.keytab.
I20260923 20:21:11.600788 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.125:34935
--webserver_interface=127.5.83.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.125.keytab
--principal=kudu/127.5.83.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:33369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:46099,127.5.83.125:34935,127.5.83.124:37389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:21:11.617672 28331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } attempt: 1
I20260923 20:21:11.618904 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:11.618429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44214)
0923 20:21:11.618644 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:11.618774 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:21:11.619932 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:11.619575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44218)
0923 20:21:11.619720 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:11.619798 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 20:21:11.620180 28331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:34935: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
I20260923 20:21:11.688447 28331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } attempt: 2
I20260923 20:21:11.689749 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:11.689272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44228)
0923 20:21:11.689517 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:11.689612 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 20:21:11.690745 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:11.690387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44232)
0923 20:21:11.690536 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:11.690613 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 20:21:11.690927 28331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:34935: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
I20260923 20:21:11.782229 28331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } attempt: 3
I20260923 20:21:11.783483 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:11.782996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44240)
0923 20:21:11.783228 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:11.783340 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 20:21:11.784749 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:11.784284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44254)
0923 20:21:11.784442 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:11.784576 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 20:21:11.784940 28331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:34935: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
I20260923 20:21:11.928233 28331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } attempt: 4
I20260923 20:21:11.929622 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:11.929011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44270)
0923 20:21:11.929342 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:11.929448 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260923 20:21:11.930737 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:11.930319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44282)
0923 20:21:11.930487 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:11.930587 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 20:21:11.930975 28331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.125:34935: Network error: Client connection negotiation failed: client connection to 127.5.83.125:34935: connect: Connection refused (error 111)
W20260923 20:21:11.949138 28338 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:21:11.949468 28338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:11.949544 28338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:11.949625 28338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 20:21:11.957072 28338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:11.957185 28338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:11.957244 28338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:11.957288 28338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:11.957351 28338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:11.967444 28338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:33369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:46099,127.5.83.125:34935,127.5.83.124:37389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.125.keytab
--principal=kudu/127.5.83.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.125:34935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.83.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:11.968926 28338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:11.970309 28338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:11.982383 28343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:11.982431 28346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:11.982465 28344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:11.983412 28338 server_base.cc:1034] running on GCE node
I20260923 20:21:11.984273 28338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:11.985862 28338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:11.987126 28338 hybrid_clock.cc:648] HybridClock initialized: now 1790194871987070 us; error 48 us; skew 500 ppm
I20260923 20:21:11.987493 28338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:11 dist-test-slave-vl24 krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194871, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:11.990988 28338 init.cc:377] Logged in from keytab as kudu/127.5.83.125@KRBTEST.COM (short username kudu)
I20260923 20:21:11.992578 28338 webserver.cc:460] Webserver started at http://127.5.83.125:33403/ using document root <none> and password file <none>
I20260923 20:21:11.993359 28338 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:11.993499 28338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:11.993846 28338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:11.997262 28338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/data/instance:
uuid: "44b4e424092d4e428f60105eee9d93fe"
format_stamp: "Formatted at 2026-09-23 20:21:11 on dist-test-slave-vl24"
server_key: "74c94aaca846f3c5f5f432b8e23b0cca"
server_key_iv: "19b86aa33f8693b3346a83a8c8be3309"
server_key_version: "encryptionkey@0"
I20260923 20:21:11.998121 28338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/wal/instance:
uuid: "44b4e424092d4e428f60105eee9d93fe"
format_stamp: "Formatted at 2026-09-23 20:21:11 on dist-test-slave-vl24"
server_key: "74c94aaca846f3c5f5f432b8e23b0cca"
server_key_iv: "19b86aa33f8693b3346a83a8c8be3309"
server_key_version: "encryptionkey@0"
I20260923 20:21:12.003891 28338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 20:21:12.007505 28353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:12.008927 28338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:12.009115 28338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/wal
uuid: "44b4e424092d4e428f60105eee9d93fe"
format_stamp: "Formatted at 2026-09-23 20:21:11 on dist-test-slave-vl24"
server_key: "74c94aaca846f3c5f5f432b8e23b0cca"
server_key_iv: "19b86aa33f8693b3346a83a8c8be3309"
server_key_version: "encryptionkey@0"
I20260923 20:21:12.009367 28338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:12.063333 28338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:12.068131 28338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:12.068427 28338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:12.087299 28338 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.125:34935
I20260923 20:21:12.087328 28404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.125:34935 every 8 connection(s)
I20260923 20:21:12.088855 28338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/data/info.pb
I20260923 20:21:12.091625 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 28338
I20260923 20:21:12.091832 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-1/wal/instance
I20260923 20:21:12.092420 5453 external_mini_cluster.cc:1428] Setting key 5ee36086826cd9efdfde1892c81126e0
I20260923 20:21:12.093633 28405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:12.104964 28405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:21:12 dist-test-slave-vl24 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194871, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.125@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.83.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.83.124@KRBTEST.COM" created.
I20260923 20:21:12.127589 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.108208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47224 (local address 127.5.83.126:46099)
0923 20:21:12.108757 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.108762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.109805 (+ 1043us) server_negotiation.cc:407] Connection header received
0923 20:21:12.111732 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.111738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.111830 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.111954 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.113704 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.115013 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.116572 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.117323 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.117468 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.121061 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.121082 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.121089 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.121156 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.124573 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.125264 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.125268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.125270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.125395 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.125843 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.125846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.125848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.126200 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.126372 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.127161 (+ 789us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.127329 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":327,"threads_started":1}
I20260923 20:21:12.129256 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.108830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46099 (local address 127.0.0.1:47224)
0923 20:21:12.109637 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.109675 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:12.111440 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:12.112203 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:12.112212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:12.112860 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:12.113438 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.113456 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.115172 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.115176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.116384 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.116396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.117525 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.117530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.117737 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.118484 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:12.118505 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:12.120820 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:12.124715 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.124722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.124734 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.125158 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.125516 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.125523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.125529 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.125742 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.126318 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:12.126324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:12.126732 (+ 408us) client_negotiation.cc:769] Sending connection context
0923 20:21:12.127068 (+ 336us) client_negotiation.cc:240] Negotiation successful
0923 20:21:12.127635 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":362,"threads_started":1}
I20260923 20:21:12.131035 28405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:21:12 dist-test-slave-vl24 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194871, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.125@KRBTEST.COM for kudu/127.5.83.125@KRBTEST.COM
I20260923 20:21:12.147310 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.131743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44286)
0923 20:21:12.131987 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.132002 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:12.132080 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:12.133456 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:12.133464 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:12.133999 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:12.134209 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.134223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.135637 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.135645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.137187 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.137205 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.138041 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.138054 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.138281 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.138951 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:12.138978 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:12.141499 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:12.144804 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.144812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.144816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.145169 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.145568 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.145573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.145577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.145703 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.146294 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:12.146300 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:12.146468 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 20:21:12.146620 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 20:21:12.146908 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260923 20:21:12.147418 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.132055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.5.83.125:34935)
0923 20:21:12.132839 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.132850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.132886 (+ 36us) server_negotiation.cc:407] Connection header received
0923 20:21:12.132950 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.132960 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.133244 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.133405 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.134385 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.135493 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.137326 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.137888 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.137986 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.141715 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.141727 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.141742 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.141768 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.144675 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.145301 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.145310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.145315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.145442 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.145803 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.145811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.145814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.146173 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.146395 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.146736 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.147034 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":564,"thread_start_us":369,"threads_started":1}
I20260923 20:21:12.149951 28405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:21:12.151086 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.150634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:59924)
0923 20:21:12.150799 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.150929 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 20:21:12.151389 28355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
I20260923 20:21:12.153221 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.152795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:59932)
0923 20:21:12.153009 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.153095 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260923 20:21:12.153585 28405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:37389: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.83.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.124.keytab.
Entry for principal kudu/127.5.83.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.124.keytab.
I20260923 20:21:12.166219 5453 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
/tmp/dist-test-taskCNgSr1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.124:37389
--webserver_interface=127.5.83.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.124.keytab
--principal=kudu/127.5.83.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.83.84:33369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.83.126:46099,127.5.83.125:34935,127.5.83.124:37389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 20:21:12.202940 28405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } attempt: 1
I20260923 20:21:12.204175 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.203722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:59936)
0923 20:21:12.203920 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.204022 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 20:21:12.205257 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.204903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:59948)
0923 20:21:12.205058 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.205152 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 20:21:12.205466 28405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:37389: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
I20260923 20:21:12.230232 28331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } attempt: 5
Sep 23 20:21:12 dist-test-slave-vl24 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194871, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for kudu/127.5.83.125@KRBTEST.COM
I20260923 20:21:12.245056 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.231017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44292)
0923 20:21:12.231249 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.231271 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:12.231423 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:12.231842 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:12.231845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:12.232220 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:12.232444 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.232453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.233962 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.233966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.234823 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.234834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.235836 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.235840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.235999 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.236677 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:12.236694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:12.238676 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:12.242649 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.242653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.242656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.243032 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.243434 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.243437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.243440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.243631 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.244262 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:12.244266 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:12.244396 (+ 130us) client_negotiation.cc:769] Sending connection context
0923 20:21:12.244517 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 20:21:12.244803 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 20:21:12.245219 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.231278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44292 (local address 127.5.83.125:34935)
0923 20:21:12.231493 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.231499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.231522 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:21:12.231606 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.231612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.231692 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.231807 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.232698 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.233819 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.234992 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.235686 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.235815 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.238872 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.238878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.238883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.238922 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.242508 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.243151 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.243157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.243161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.243317 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.243756 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.243760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.243763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.244136 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.244296 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.244690 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.244920 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 20:21:12.246300 28331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:21:12.247291 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.246927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:59956)
0923 20:21:12.247081 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.247154 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 20:21:12.248214 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.247877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:59960)
0923 20:21:12.248029 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.248113 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 20:21:12.248410 28331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:37389: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
I20260923 20:21:12.273656 28405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } attempt: 2
I20260923 20:21:12.274719 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.274288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:59966)
0923 20:21:12.274476 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.274569 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 20:21:12.275672 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.275318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:59970)
0923 20:21:12.275449 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.275533 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 20:21:12.275854 28405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:37389: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
I20260923 20:21:12.299646 28331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } attempt: 1
I20260923 20:21:12.300863 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.300355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:59976)
0923 20:21:12.300604 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.300700 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 20:21:12.301863 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.301555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:59982)
0923 20:21:12.301701 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.301764 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 20:21:12.302073 28331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:37389: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
I20260923 20:21:12.367077 28405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } attempt: 3
I20260923 20:21:12.368516 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.367810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:59984)
0923 20:21:12.368115 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.368264 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 20:21:12.370124 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.369366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:59988)
0923 20:21:12.369810 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.369933 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260923 20:21:12.370301 28331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } attempt: 2
W20260923 20:21:12.370447 28405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:37389: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
I20260923 20:21:12.371347 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.370849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:60000)
0923 20:21:12.371071 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.371171 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 20:21:12.372346 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.372032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:60004)
0923 20:21:12.372173 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.372245 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 20:21:12.372584 28331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:37389: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
I20260923 20:21:12.478796 28331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } attempt: 3
I20260923 20:21:12.479924 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.479471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:60020)
0923 20:21:12.479664 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.479754 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 20:21:12.481042 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.480703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:60024)
0923 20:21:12.480871 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.480945 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 20:21:12.481230 28331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:37389: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
W20260923 20:21:12.512409 28413 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 20:21:12.512876 28413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 20:21:12.513015 28413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 20:21:12.513173 28413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 20:21:12.513629 28405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } attempt: 4
I20260923 20:21:12.514705 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.514284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:60030)
0923 20:21:12.514469 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.514554 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 20:21:12.515632 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.515307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:60046)
0923 20:21:12.515471 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.515534 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 20:21:12.515851 28405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.83.124:37389: Network error: Client connection negotiation failed: client connection to 127.5.83.124:37389: connect: Connection refused (error 111)
W20260923 20:21:12.521232 28413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 20:21:12.521355 28413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 20:21:12.521476 28413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 20:21:12.521560 28413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 20:21:12.521656 28413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 20:21:12.531903 28413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.83.84:33369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.83.126:46099,127.5.83.125:34935,127.5.83.124:37389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/krb5kdc/kudu_127.5.83.124.keytab
--principal=kudu/127.5.83.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.83.124:37389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.83.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:12:23 UTC on ccycloud.quasar-hfwmlo.root.comops.site
build id 4668084
ASAN enabled
I20260923 20:21:12.533238 28413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 20:21:12.534508 28413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 20:21:12.546491 28419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:12.546444 28421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 20:21:12.546989 28418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 20:21:12.547435 28413 server_base.cc:1034] running on GCE node
I20260923 20:21:12.548269 28413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 20:21:12.549794 28413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 20:21:12.551050 28413 hybrid_clock.cc:648] HybridClock initialized: now 1790194872550958 us; error 55 us; skew 500 ppm
I20260923 20:21:12.551415 28413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 20:21:12 dist-test-slave-vl24 krb5kdc[28250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194872, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 20:21:12.554618 28413 init.cc:377] Logged in from keytab as kudu/127.5.83.124@KRBTEST.COM (short username kudu)
I20260923 20:21:12.556126 28413 webserver.cc:460] Webserver started at http://127.5.83.124:37803/ using document root <none> and password file <none>
I20260923 20:21:12.556918 28413 fs_manager.cc:359] Metadata directory not provided
I20260923 20:21:12.557091 28413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 20:21:12.557508 28413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 20:21:12.560901 28413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/data/instance:
uuid: "28131c89c5194c4997b63678ba747a4b"
format_stamp: "Formatted at 2026-09-23 20:21:12 on dist-test-slave-vl24"
server_key: "1601e0f755956324c36cee4729577fba"
server_key_iv: "2db393fa84f2b4f246a867eefd9ecf13"
server_key_version: "encryptionkey@0"
I20260923 20:21:12.561883 28413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/wal/instance:
uuid: "28131c89c5194c4997b63678ba747a4b"
format_stamp: "Formatted at 2026-09-23 20:21:12 on dist-test-slave-vl24"
server_key: "1601e0f755956324c36cee4729577fba"
server_key_iv: "2db393fa84f2b4f246a867eefd9ecf13"
server_key_version: "encryptionkey@0"
I20260923 20:21:12.568668 28413 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 20:21:12.572322 28428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 20:21:12.573880 28413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 20:21:12.574074 28413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/data,/tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/wal
uuid: "28131c89c5194c4997b63678ba747a4b"
format_stamp: "Formatted at 2026-09-23 20:21:12 on dist-test-slave-vl24"
server_key: "1601e0f755956324c36cee4729577fba"
server_key_iv: "2db393fa84f2b4f246a867eefd9ecf13"
server_key_version: "encryptionkey@0"
I20260923 20:21:12.574299 28413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 20:21:12.600521 28413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 20:21:12.604936 28413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 20:21:12.605276 28413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 20:21:12.624137 28413 rpc_server.cc:307] RPC server started. Bound to: 127.5.83.124:37389
I20260923 20:21:12.624157 28480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.83.124:37389 every 8 connection(s)
I20260923 20:21:12.625573 28413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/data/info.pb
I20260923 20:21:12.625921 5453 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu as pid 28413
I20260923 20:21:12.626101 5453 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCNgSr1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790194506409421-5453-0/minicluster-data/master-2/wal/instance
I20260923 20:21:12.626669 5453 external_mini_cluster.cc:1428] Setting key 3c2bcadd7fbf490ee946c46d037d5590
I20260923 20:21:12.629941 28481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 20:21:12.638223 28481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 20:21:12 dist-test-slave-vl24 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.124@KRBTEST.COM
Sep 23 20:21:12 dist-test-slave-vl24 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194872, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.124@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
I20260923 20:21:12.650701 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.628808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.5.83.124:37389)
0923 20:21:12.629646 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.629657 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.629697 (+ 40us) server_negotiation.cc:407] Connection header received
0923 20:21:12.631328 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.631350 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.631728 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.632057 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.633857 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.635179 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.637047 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.637702 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.637857 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.643209 (+ 5352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.643217 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.643238 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.643281 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.646581 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.647307 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.647314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.647320 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.647477 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.647879 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.647893 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.647895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.648416 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.648765 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.649205 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.649695 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":374,"threads_started":1}
I20260923 20:21:12.656630 28486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.641613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.5.83.126:46099)
0923 20:21:12.642766 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.642772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.642790 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:21:12.642841 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.642844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.642911 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.643035 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.644295 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.645160 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.646585 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.646980 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.647112 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.650358 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.650363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.650367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.650401 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.653596 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.654495 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.654499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.654502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.654645 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.655041 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.655048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.655056 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.655350 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.655509 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.655939 (+ 430us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.656149 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1053,"thread_start_us":927,"threads_started":1}
I20260923 20:21:12.656633 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.641483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.126:46099 (local address 127.0.0.1:47226)
0923 20:21:12.642280 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.642321 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:12.642465 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:12.643091 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:12.643097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:12.643855 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:12.644117 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.644126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.645327 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.645341 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.646418 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.646429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.647171 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.647178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.647348 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.647955 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:12.647972 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:12.650158 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:12.653732 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.653740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.653745 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.654339 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.654768 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.654772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.654776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.654917 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.655480 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:12.655488 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:12.655673 (+ 185us) client_negotiation.cc:769] Sending connection context
0923 20:21:12.655854 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 20:21:12.656149 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":328,"threads_started":1}
I20260923 20:21:12.658110 28481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:21:12.658409 28331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } attempt: 4
Sep 23 20:21:12 dist-test-slave-vl24 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.126@KRBTEST.COM
Sep 23 20:21:12 dist-test-slave-vl24 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194871, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.126@KRBTEST.COM for kudu/127.5.83.124@KRBTEST.COM
Sep 23 20:21:12 dist-test-slave-vl24 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194872, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.124@KRBTEST.COM for kudu/127.5.83.125@KRBTEST.COM
I20260923 20:21:12.672861 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.658948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:60066)
0923 20:21:12.659273 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.659291 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:12.659391 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:12.659881 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:12.659886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:12.660277 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:12.660411 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.660419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.661574 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.661581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.662480 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.662491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.663253 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.663259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.663482 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.664111 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:12.664135 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:12.666779 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:12.670184 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.670190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.670194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.670597 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.670965 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.670970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.670973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.671167 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.671809 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:12.671813 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:12.671993 (+ 180us) client_negotiation.cc:769] Sending connection context
0923 20:21:12.672139 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 20:21:12.672452 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260923 20:21:12.673705 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.654066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.5.83.126:46099)
0923 20:21:12.654732 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.654737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.654756 (+ 19us) server_negotiation.cc:407] Connection header received
0923 20:21:12.654795 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.654798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.654850 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.654975 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.656222 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.657430 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.659099 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.659779 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.659927 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.666231 (+ 6304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.666238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.666243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.666295 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.669830 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.670851 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.670859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.670865 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.671041 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.671470 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.671475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.671479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.671796 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.672059 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.673216 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.673383 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":314,"threads_started":1}
I20260923 20:21:12.675693 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.658976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.5.83.125:34935)
0923 20:21:12.659201 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.659210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.659455 (+ 245us) server_negotiation.cc:407] Connection header received
0923 20:21:12.659691 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.659698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.659779 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.659890 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.660930 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.661603 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.663466 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.664116 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.664253 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.667998 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.668007 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.668013 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.668058 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.671777 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.672745 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.672751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.672754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.673091 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.673819 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.673827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.673833 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.674396 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.674543 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.675092 (+ 549us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.675311 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 20:21:12.677384 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.658914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.125:34935 (local address 127.0.0.1:44298)
0923 20:21:12.659361 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.659382 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:12.659548 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:12.660087 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:12.660092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:12.660576 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:12.660787 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.660795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.661737 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.661745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.663306 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.663323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.664295 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.664305 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.664502 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.665078 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:12.665105 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:12.667550 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:12.671916 (+ 4366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.671920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.671924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.672513 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.673308 (+ 795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.673339 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.673343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.673606 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.674629 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:12.674635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:12.674807 (+ 172us) client_negotiation.cc:769] Sending connection context
0923 20:21:12.676387 (+ 1580us) client_negotiation.cc:240] Negotiation successful
0923 20:21:12.676852 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":28}
I20260923 20:21:12.678797 28481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } has no permanent_uuid. Determining permanent_uuid...
I20260923 20:21:12.682119 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.659427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60066 (local address 127.5.83.124:37389)
0923 20:21:12.659574 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.659582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.659598 (+ 16us) server_negotiation.cc:407] Connection header received
0923 20:21:12.659641 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.659648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.659712 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.659821 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.660580 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.661442 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.662623 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.663111 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.663240 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.666948 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.666954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.666958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.666998 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.669821 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.670722 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.670729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.670737 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.670849 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.671299 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.671306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.671310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.671682 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.671840 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.674696 (+ 2856us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.675305 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
Sep 23 20:21:12 dist-test-slave-vl24 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.83.125@KRBTEST.COM
I20260923 20:21:12.687813 28331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb: Bootstrap starting.
I20260923 20:21:12.691952 28331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:12.693267 28331 log.cc:824] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:12.693769 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.675532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.5.83.125:34935)
0923 20:21:12.675933 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.675939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.675956 (+ 17us) server_negotiation.cc:407] Connection header received
0923 20:21:12.676006 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.676011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.676103 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.676230 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.677438 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.678393 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.680010 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.680772 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.680888 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.687005 (+ 6117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.687012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.687016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.687057 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.690840 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.691520 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.691527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.691530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.691697 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.692148 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.692156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.692159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.692519 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.692863 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.693156 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.693393 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":13410,"threads_started":1}
Sep 23 20:21:12 dist-test-slave-vl24 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194872, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.124@KRBTEST.COM for kudu/127.5.83.124@KRBTEST.COM
I20260923 20:21:12.699043 5453 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 20:21:12.699549 28331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb: No bootstrap required, opened a new log
I20260923 20:21:12.702270 28305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "31102c333ecb49059450bae419a179bb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47230
I20260923 20:21:12.704922 28485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.679573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:60074)
0923 20:21:12.679754 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.679774 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:12.679871 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:12.680790 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:12.680797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:12.681467 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:12.681713 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.681727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.692401 (+ 10674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.692408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.693758 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.693775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.695053 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.695059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.695279 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.695955 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:12.695987 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:12.698636 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:12.702324 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.702332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.702336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.702713 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.703089 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.703094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.703098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.703225 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.703757 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:12.703771 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:12.703918 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 20:21:12.704079 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 20:21:12.704443 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":3}
I20260923 20:21:12.704988 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.679847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.5.83.124:37389)
0923 20:21:12.680427 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.680432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.680455 (+ 23us) server_negotiation.cc:407] Connection header received
0923 20:21:12.680498 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.680503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.680601 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.680706 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.681934 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.692243 (+ 10309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.693939 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.694821 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.695009 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.698842 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.698848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.698854 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.698894 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.702191 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.702835 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.702843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.702847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.702977 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.703327 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.703334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.703337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.703635 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.703781 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.704174 (+ 393us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.704591 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":182,"server-negotiator.queue_time_us":458,"thread_start_us":306,"threads_started":1}
I20260923 20:21:12.709447 28331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } }
I20260923 20:21:12.709848 28331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31102c333ecb49059450bae419a179bb, State: Initialized, Role: FOLLOWER
I20260923 20:21:12.710368 28331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } }
I20260923 20:21:12.710803 28481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b: Bootstrap starting.
I20260923 20:21:12.715934 28481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:12.717788 28481 log.cc:824] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b: Log is configured to *not* fsync() on all Append() calls
I20260923 20:21:12.718544 28491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } } }
I20260923 20:21:12.719024 28491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:12.719930 28331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:12.720072 28305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 20:21:12.720321 28305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:12.722223 28481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b: No bootstrap required, opened a new log
I20260923 20:21:12.723832 28305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } }
I20260923 20:21:12.725672 28305 leader_election.cc:288] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [CANDIDATE]: Term 1 election: Requested vote from peers 44b4e424092d4e428f60105eee9d93fe (127.5.83.125:34935), 28131c89c5194c4997b63678ba747a4b (127.5.83.124:37389)
I20260923 20:21:12.741737 28481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } }
I20260923 20:21:12.742197 28481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28131c89c5194c4997b63678ba747a4b, State: Initialized, Role: FOLLOWER
I20260923 20:21:12.742705 28481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } }
W20260923 20:21:12.745779 28503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 20:21:12.745898 28503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 20:21:12.742695 28263 master_runner.cc:186] Error getting master registration for 127.5.83.125:34935: OK, instance_id { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" instance_seqno: 1790194872081819 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:21:12.758842 28263 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:21:12.759505 28508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } } }
I20260923 20:21:12.759959 28508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:12.760857 28481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:12.761022 28455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31102c333ecb49059450bae419a179bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "28131c89c5194c4997b63678ba747a4b"
I20260923 20:21:12.761709 28455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [term 0 FOLLOWER]: Advancing to term 1
W20260923 20:21:12.761905 28284 tablet.cc:2369] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:21:12.765494 28455 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31102c333ecb49059450bae419a179bb in term 1.
I20260923 20:21:12.766395 28280 leader_election.cc:302] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28131c89c5194c4997b63678ba747a4b, 31102c333ecb49059450bae419a179bb; no voters:
I20260923 20:21:12.766876 28491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 20:21:12.777731 28491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [term 1 LEADER]: Becoming Leader. State: Replica: 31102c333ecb49059450bae419a179bb, State: Running, Role: LEADER
I20260923 20:21:12.779064 28491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } }
I20260923 20:21:12.794690 28514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31102c333ecb49059450bae419a179bb. Latest consensus state: current_term: 1 leader_uuid: "31102c333ecb49059450bae419a179bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } } }
I20260923 20:21:12.795624 28514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:12.796484 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.751525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.5.83.126:46099)
0923 20:21:12.751749 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.751758 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.765453 (+ 13695us) server_negotiation.cc:407] Connection header received
0923 20:21:12.765782 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.765788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.765885 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.765977 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.769389 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.770860 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.776068 (+ 5208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.777107 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.777250 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.786520 (+ 9270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.786535 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.786543 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.786606 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.791912 (+ 5306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.793153 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.793163 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.793174 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.793378 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.793961 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.793969 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.793975 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.794490 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.794820 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.795618 (+ 798us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.795961 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 20:21:12.809121 28517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 20:21:12.811355 28517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 20:21:12.815034 28405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } attempt: 5
I20260923 20:21:12.827873 28489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.797906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.5.83.124:37389)
0923 20:21:12.798137 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.798143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.800401 (+ 2258us) server_negotiation.cc:407] Connection header received
0923 20:21:12.800743 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.800752 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.800860 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.800946 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.802960 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.804451 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.807329 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.808283 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.808407 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.813597 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.813606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.813614 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.813668 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.818470 (+ 4802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.824611 (+ 6141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.824619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.824626 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.824822 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.825359 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.825367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.825373 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.825864 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.826163 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.826933 (+ 770us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.827352 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 20:21:12 dist-test-slave-vl24 krb5kdc[28250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790194871, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.83.125@KRBTEST.COM for kudu/127.5.83.124@KRBTEST.COM
I20260923 20:21:12.835130 28455 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [term 1 FOLLOWER]: Refusing update from remote peer 31102c333ecb49059450bae419a179bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:12.836611 28491 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:12.845108 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.815632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.83.124:37389 (local address 127.0.0.1:60086)
0923 20:21:12.816036 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0923 20:21:12.816057 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 20:21:12.816168 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 20:21:12.817257 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 20:21:12.817274 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 20:21:12.817755 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0923 20:21:12.817953 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.817965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.819500 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.819509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.821565 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 20:21:12.821587 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.823163 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.823173 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 20:21:12.823506 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.825417 (+ 1911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 20:21:12.825463 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 20:21:12.829497 (+ 4034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 20:21:12.836570 (+ 7073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.836577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.836583 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.837853 (+ 1270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.842594 (+ 4741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 20:21:12.842600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 20:21:12.842604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 20:21:12.842894 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 20:21:12.843860 (+ 966us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 20:21:12.843865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 20:21:12.844117 (+ 252us) client_negotiation.cc:769] Sending connection context
0923 20:21:12.844347 (+ 230us) client_negotiation.cc:240] Negotiation successful
0923 20:21:12.844709 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268}
I20260923 20:21:12.845155 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.816594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.5.83.124:37389)
0923 20:21:12.816779 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.816785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.816803 (+ 18us) server_negotiation.cc:407] Connection header received
0923 20:21:12.816849 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.816854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.816927 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.817026 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.818152 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.819314 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.821795 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.822732 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.822878 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.829775 (+ 6897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.829784 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.829792 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.829844 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.834364 (+ 4520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.842155 (+ 7791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.842161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.842165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.842347 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.843100 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.843106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.843109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.843559 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.843732 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.844592 (+ 860us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.844824 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 20:21:12.847589 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:12.765667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44322 (local address 127.5.83.125:34935)
0923 20:21:12.765848 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:12.765855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:12.789571 (+ 23716us) server_negotiation.cc:407] Connection header received
0923 20:21:12.790536 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:12.790543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:12.790642 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:12.790722 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:12.805384 (+ 14662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.806515 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.811093 (+ 4578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:12.811765 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:12.811870 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:12.820188 (+ 8318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 20:21:12.820198 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 20:21:12.820206 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 20:21:12.820290 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0923 20:21:12.825075 (+ 4785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.830427 (+ 5352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.830433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.830436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.830598 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 20:21:12.831386 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 20:21:12.831392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 20:21:12.831395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 20:21:12.831758 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 20:21:12.832006 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:12.846929 (+ 14923us) server_negotiation.cc:299] Negotiation successful
0923 20:21:12.847213 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 20:21:12.851372 28405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe: Bootstrap starting.
I20260923 20:21:12.855288 28405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe: Neither blocks nor log segments found. Creating new log.
I20260923 20:21:12.856631 28405 log.cc:824] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe: Log is configured to *not* fsync() on all Append() calls
W20260923 20:21:12.857405 28434 tablet.cc:2369] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 20:21:12.868067 28405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe: No bootstrap required, opened a new log
I20260923 20:21:12.868403 28508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31102c333ecb49059450bae419a179bb. Latest consensus state: current_term: 1 leader_uuid: "31102c333ecb49059450bae419a179bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } } }
I20260923 20:21:12.868749 28508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:12.870168 28519 mvcc.cc:205] Tried to move back new op lower bound from 7332638199103766528 to 7332638198914269184. Current Snapshot: MvccSnapshot[applied={T|T < 7332638199103766528}]
I20260923 20:21:12.881948 28405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } }
I20260923 20:21:12.882665 28405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b4e424092d4e428f60105eee9d93fe, State: Initialized, Role: FOLLOWER
I20260923 20:21:12.883339 28405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } }
I20260923 20:21:12.890926 28522 mvcc.cc:205] Tried to move back new op lower bound from 7332638199103766528 to 7332638198914269184. Current Snapshot: MvccSnapshot[applied={T|T < 7332638199103766528}]
I20260923 20:21:12.890884 28532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } } }
I20260923 20:21:12.893158 28405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [sys.catalog]: configured and running, proceeding with master startup.
I20260923 20:21:12.893402 28379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [term 0 FOLLOWER]: Advancing to term 1
I20260923 20:21:12.893878 28532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:12.896162 28508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31102c333ecb49059450bae419a179bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } } }
I20260923 20:21:12.896440 28508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:12.896628 28379 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [term 1 FOLLOWER]: Refusing update from remote peer 31102c333ecb49059450bae419a179bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 20:21:12.897670 28514 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 20:21:12.893517 28380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31102c333ecb49059450bae419a179bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "44b4e424092d4e428f60105eee9d93fe"
I20260923 20:21:12.899930 28517 catalog_manager.cc:1347] Generated new cluster ID: 54491d95a12b460b9f5295b2ea33196b
I20260923 20:21:12.900022 28517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 20:21:12.900859 28514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31102c333ecb49059450bae419a179bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } } }
I20260923 20:21:12.901136 28514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:12.902004 28380 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31102c333ecb49059450bae419a179bb in term 1.
I20260923 20:21:12.919010 28517 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 20:21:12.921277 28517 catalog_manager.cc:1504] Loading token signing keys...
I20260923 20:21:12.906205 28413 master_runner.cc:186] Error getting master registration for 127.5.83.125:34935: OK, instance_id { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" instance_seqno: 1790194872081819 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 20:21:12.940081 28517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb: Generated new TSK 0
I20260923 20:21:12.941294 28517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 20:21:12.949038 28413 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 20:21:12.963662 28529 catalog_manager.cc:1259] Loaded cluster ID: 54491d95a12b460b9f5295b2ea33196b
I20260923 20:21:12.965423 28529 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b: loading cluster ID for follower catalog manager: success
I20260923 20:21:12.969470 28529 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b: acquiring CA information for follower catalog manager: success
I20260923 20:21:12.981199 28529 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 28131c89c5194c4997b63678ba747a4b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 20:21:12.985324 28543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31102c333ecb49059450bae419a179bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } } }
I20260923 20:21:12.985736 28543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:12.985597 28491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31102c333ecb49059450bae419a179bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } } }
I20260923 20:21:12.985900 28491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31102c333ecb49059450bae419a179bb [sys.catalog]: This master's current role is: LEADER
I20260923 20:21:12.986073 28532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31102c333ecb49059450bae419a179bb. Latest consensus state: current_term: 1 leader_uuid: "31102c333ecb49059450bae419a179bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31102c333ecb49059450bae419a179bb" member_type: VOTER last_known_addr { host: "127.5.83.126" port: 46099 } } peers { permanent_uuid: "44b4e424092d4e428f60105eee9d93fe" member_type: VOTER last_known_addr { host: "127.5.83.125" port: 34935 } } peers { permanent_uuid: "28131c89c5194c4997b63678ba747a4b" member_type: VOTER last_known_addr { host: "127.5.83.124" port: 37389 } } }
I20260923 20:21:12.986334 28532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe [sys.catalog]: This master's current role is: FOLLOWER
I20260923 20:21:13.028128 28551 catalog_manager.cc:1259] Loaded cluster ID: 54491d95a12b460b9f5295b2ea33196b
I20260923 20:21:13.028293 28551 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe: loading cluster ID for follower catalog manager: success
I20260923 20:21:13.032513 28551 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe: acquiring CA information for follower catalog manager: success
I20260923 20:21:13.035375 28551 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44b4e424092d4e428f60105eee9d93fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 20:21:13.134614 28487 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:13.108187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.5.83.126:46099)
0923 20:21:13.109040 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:13.109049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:13.127011 (+ 17962us) server_negotiation.cc:407] Connection header received
0923 20:21:13.132745 (+ 5734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:13.132756 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:13.132894 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:13.133007 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:13.134314 (+ 1307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47244: BlockingRecv error: recv got EOF from 127.0.0.1:47244 (error 108)
Metrics: {"server-negotiator.queue_time_us":680}
W20260923 20:21:13.137941 28410 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:13.127643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.5.83.125:34935)
0923 20:21:13.127839 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:13.127846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:13.136311 (+ 8465us) server_negotiation.cc:407] Connection header received
0923 20:21:13.136499 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:13.136505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:13.136615 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:13.136733 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:13.137715 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44332: BlockingRecv error: recv got EOF from 127.0.0.1:44332 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 20:21:13.140157 28484 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 20:21:13.137789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.5.83.124:37389)
0923 20:21:13.137985 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:13.137991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:13.138857 (+ 866us) server_negotiation.cc:407] Connection header received
0923 20:21:13.139019 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:13.139026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:13.139104 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:13.139204 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 20:21:13.139922 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60102: BlockingRecv error: recv got EOF from 127.0.0.1:60102 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 20:21:13.173144 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:13.144121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.5.83.125:34935)
0923 20:21:13.144294 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:13.144301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:13.161384 (+ 17083us) server_negotiation.cc:407] Connection header received
0923 20:21:13.161666 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:13.161672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:13.161890 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:13.162081 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:13.162617 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:13.163760 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:13.166100 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:13.166786 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:13.166900 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:13.171685 (+ 4785us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:13.172438 (+ 753us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:13.172622 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:13.172693 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 20:21:13.172783 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 20:21:13.227005 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:13.218712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.5.83.124:37389)
0923 20:21:13.218902 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:13.218910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:13.219141 (+ 231us) server_negotiation.cc:407] Connection header received
0923 20:21:13.219264 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:13.219270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:13.219493 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:13.219686 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:13.220413 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:13.221565 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:13.223766 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:13.224483 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:13.224617 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:13.225452 (+ 835us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:13.226150 (+ 698us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:13.226277 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:13.226541 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 20:21:13.226637 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 20:21:13.269158 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:13.250309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.5.83.126:46099)
0923 20:21:13.250518 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:13.250526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:13.261220 (+ 10694us) server_negotiation.cc:407] Connection header received
0923 20:21:13.261610 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:13.261617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:13.261876 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:13.262022 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:13.262829 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:13.264057 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:13.266268 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:13.266971 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:13.267077 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:13.267851 (+ 774us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:13.268343 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:13.268447 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:13.268711 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 20:21:13.268807 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 20:21:13.279721 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 28263
I20260923 20:21:13.281597 28325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:13.282686 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:13.274143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.5.83.124:37389)
0923 20:21:13.274353 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:13.274359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:13.274379 (+ 20us) server_negotiation.cc:407] Connection header received
0923 20:21:13.274451 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:13.274457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:13.274542 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:13.274642 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:13.275224 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:13.276360 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:13.280047 (+ 3687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:13.280747 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:13.280842 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:13.281913 (+ 1071us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:13.282030 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:13.282088 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:13.282316 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 20:21:13.282402 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 20:21:13.282691 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 20:21:13.262146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.5.83.125:34935)
0923 20:21:13.262376 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 20:21:13.262382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 20:21:13.270114 (+ 7732us) server_negotiation.cc:407] Connection header received
0923 20:21:13.270387 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 20:21:13.270393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 20:21:13.270505 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 20:21:13.270572 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 20:21:13.271154 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:13.272320 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:13.278131 (+ 5811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 20:21:13.278886 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 20:21:13.278995 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 20:21:13.281860 (+ 2865us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 20:21:13.282016 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 20:21:13.282088 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 20:21:13.282321 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 20:21:13.282417 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 20:21:13.409795 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 28263
I20260923 20:21:13.432056 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 28338
I20260923 20:21:13.433233 28400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:13.539891 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 28338
I20260923 20:21:13.559113 5453 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu pid 28413
I20260923 20:21:13.560240 28476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 20:21:13.669142 5453 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCNgSr1/build/asan/bin/kudu with pid 28413
2026-09-23T20:21:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4038 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9145 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (367173 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST